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EXECUTIVE  SUMMARY 


This  report  summarizes  an  examination  of  the  technologies  employed  in 
acquiring  three-dimensional  Information  for  robotlc(s)  applications.  Of 
specific  interest  is  the  Identification  of  technology  concepts/ideas  that  have 
significant  promise  for  improving  abilities  to  acquire  such  information. 


It  became  apparent  during  this  study  that  acquiring  three-dimensional 
information  for  robotics  application  can  be  usage-dependent.  We  have 
attempted  to  generalize  this  review  and  the  conclusions  reached.  However, 
each  prospective  application  should  be  carefully  examined  to  Identify  the 
unique  operating  conditions  or  constraints  which  might  be  utilized  to  simplify 
the  acquisition  of  three-dimensional  information.  In  fact,  at  current  per¬ 
formance  levels  of  the  state-of-the-art,  the  quantities  of  data  associated 
with  detailed  three-dimensional  information  probably  could  not  be  effectively 
utilized.  Imaging  systems,  although  potentially  capable  of  considerably 
enhancing  robot  performance,  are  expensive.  The  intelligent  system  designer 
should  consider  performing  a  trade-off  between  the  dollars  available  and  the 
acquisition  of  enough  Imaging  capability  to  assure  the  efficient  and  timely 
completion  of  an  objective.  A  proper  trade-off  allows  consideration  to  be 
given  whether  expansion  capabillties/capacitles  can  be  built  into  the  system. 


The  technologies  considered  in  this  review  Include 

•  optical  stereoscopy, 

•  proximity  sensing, 

•  laser  scanning,  and 

•  structured  light. 
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In  addition  to  surveying  recent  literature,  facilities  and  researchers 
actively  engaged  in  researching  these  technologies  were  contacted  and  queried. 
The  information  provided  was  examined  with  respect  to  known  and  anticipated 
requirements.  Recommendations  are  made  for  both  advanced  research  and 
extended  efforts  in  the  following  general  areas. 

Optical  Stereoscopy 

Based  on  human  vision  and  application  of  relatively  simple  triangulation 
theory,  stereoscopy  is  receiving  considerable  attention  for  use  in  acquiring 
three-dimensional  information.  The  most  significant  roadblock  to  effectively 
using  stereoscopy  in  robotics  is  the  problem  of  correlating  two  images  to 
uniquely  identify  the  same  point  in  each  image.  The  correlation  problem  is 
actively  being  addressed  from  several  directions,  including 

•  edge  and  vertex  enhancement, 

•  grey-scale  correlation,  and 

•  shape  correlation. 

One  avenue  of  approach  not  currently  being  addressed  is  the  added  use  of  color 
as  a  discriminant  In  aiding  Image  correlation.  Recent  advances  In  acquiring 
color  data  with  solid  state  image  sensors  add  to  the  potential  utility  of  the 
concept.  Specifically,  a  trade-off  study  of  grey-scale  digitization  levels 
versus  color  digitization  levels  should  be  undertaken. 

Although  Improved  solid  state  imaging  capability  is  desirable  for 
stereoscopy,  the  state-of-the-art  in  data  collection  is  generally  ahead  of 
current  abilities  to  effectively  use  the  data  generated.  The  commercial 
Imaging  industry  (for  TV,  cameras,  etc.)  potentially  represents  a  much  greater 
driving  force  than  robotics  application  for  generating  Improvements  in  image 
sensor  capabilities.  However,  some  of  the  image  correlation  techniques 
presently  being  studied  should  potentially  be  implemented  in  hardware. 
Consideration  should  be  given  to  coupling  the  image  correlation  techniques 
directly  on  the  Imaging  sensor.  Functional  combinations,  such  as  chemical 
sensors  and  microelectronics,  are  becoming  more  common  and  could  be  useful 
here. 
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Proximity  Sensing 

Although  not  three-dimensional,  proximity  sensing  remains  a  considerable 
problem  for  robotics.  Generally,  as  a  manipulator,  or  perhaps  even  a  moving 
robot  itself,  approaches  an  object  under  consideration,  the  usefulness  of 
certain  sensor  systems  greatly  diminishes.  This  can  be  caused  by  obstructed 
views  and/or  inherent  sensor  limitations.  The  development  of  auxiliary 
proximity  sensing  techniques  appears  highly  desirable.  Recommended  for  more 
detailed  study  in  proximity  sensing  applications  are  the  use  of  fiber  optic 
sensors  and  ultrasonic  probing  concepts  and  techniques. 

Laser  Scanning 

The  applications  of  laser  techniques  have  been  identified  as  one  of  the 
technologies  exhibiting  the  greatest  promise  for  versatile,  three-dimensional 
data  acquisition.  Lasers  can  be  used  to  acquire  three-dimensional  coordinate 
information  in  two  distinctly  different  approaches.  In  one  approach,  ranging 
information  is  obtained  from  tlme-of-fl ight  measurements;  in  the  second,  the 
unique  character  of  the  laser  is  used  to  generate  a  controlled  illumination 
pattern  which  permits  the  acquisition  of  ranging  information  using  simple 
triangulation. 

Several  ranging  systems  which  have  been  formulated  and  used  have  shown 
great  promise.  For  Implementation  In  robotics,  additional  work  is  required  in 
several  areas.  In  the  first  application,  data  acquisition  rates  and  signal- 
to-noise  ratios  could  be  Improved  with  the  development  of  higher  power  semi¬ 
conductor  lasers  (preferably  without  cooling  requirements).  Semiconductor 
lasers  are  emphasized  because  of  an  inherent  ruggedness  and  also  because  they 
are  generally  smaller  and  easier  to  handle.  Second,  Improved  means  for 
nonmechanical  laser  beam  deflection  must  be  developed.  Rotating/oscillating 
mirrors  and/or  prisms  can  perform  the  function,  but  they  lack  the  ruggedness 
required  for  field  or  factory  use.  Acousto-optic  deflection  technology  has 
use  where  beam  deflection  is  extremely  limited,  but  it  cannot  be  used  for  the 
larger  deflections  desired  for  laser  ranging  (in  principle,  they  could  be 
serially  used,  but  the  resultant  beam  degradation  and  added  computational 
complexity  make  such  use  impractical). 
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Structured  Light 

Utilizing  a  controlled  laser  illumination  source  with  simple  triangula¬ 
tion  is  a  promising  concept.  The  most  advanced  of  these  concepts  generates  an 
illuminating  matrix  of  laser  spots  to  be  viewed  with  passive  imaging  tech¬ 
nology.  Presently,  the  most  serious  drawbacks  to  this  technique  appear  to  be 
array  coding  and  the  large  quantity  of  data  which  must  be  stored  and  manipu¬ 
lated  to  generate  the  desired  range  information.  There  is  an  improvement  in 
the  amount  of  data  over  that  required  for  the  image  correlation  needed  to 
perform  stereoscopic  analysis,  but  the  numbers  are  still  large.  Improvements 
are  needed. 

With  the  recommendations  given  above,  we  would  like  to  add  one  general 
observation.  There  are  inherent  strengths  in  the  laser-scanning  approach 
which  should  eventually  be  enhanced  by  advances  in  holographic  techniques  and 
in  data  storage,  processing,  and  retrieval.  It  is  not  clear,  at  present, 
exactly  what  form  such  a  hybrid  system  might  take,  but  current  research  to 
develop  real-time,  erasable  holographic  storage  elements  should  find  applica¬ 
tion  here. 
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1.  INTRODUCTION 


A  robotic  system  can  be  described  as  one  capable  of  receiving  communica¬ 
tions,  understanding  its  environment,  formulating  and  executing  “plans,"  and 
monitoring  its  actions.  Although  both  the  capabilities  and  sales  of  “robots" 
show  extremely  sharp  growth  rates,  only  a  limited  number  of  systems  are 
capable  of  performing  all  of  the  elements  outlined  above.  Robots  are  finding 
increased  utilization  in  applications  too  tedious,  dangerous,  and  precise  for 
human  execution,  and  are  proving  to  be  more  reliable,  less  demanding,  and  more 
cost-effective  than  human  labor  in  many  manufacturing  applications.  Increased 
robotics  utilization  is  pushing  their  use  into  exploration  and  to  other 
applications  requiring  decision-making  capabilities. 

Two  of  the  robotic  capabilities  outlined  above  require  the  use  of  sensory 
systems  to  acquire  data  external  to  the  robot.  In  both  understanding  its 
environment  and  monitoring  its  actions,  a  robotic  system  is  dependent  on 
sensors  to  probe  and  quantify  the  external  environment.  The  sensors  must 
accomplish  these  tasks  accurately  and  rapidly. 

Current  sensor  systems  have  limited  abilities  to  acquire  such  informa¬ 
tion,  and  the  primary  technology  employed  (excepting  tactile  sensing)  is 
two-dimensional  imaging  using  conventional  optical  systems.  Using  some  of  the 
concepts  described  later  in  this  report,  the  ability  to  acquire,  process,  and 
utilize  two-dimensional  information  has  been  extended  to  permit  the  acquisi¬ 
tion  and  use  of  limited  amounts  of  three-dimensional  information.  However, 
current  abilities  to  directly  acquire  three-dimensional  information  are 
minimal.  This  study  was  undertaken  to  review  sensory  systems  and  techniques 
for  the  purpose  of  identifying  concepts  and/or  ideas  having  the  potential 
to  significantly  enhance  abilities  to  acquire  three-dimensional  range 
information. 
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The  acquisition  of  three-dimensional  information  for  application  with 
robotic  systems  can  be  referred  to  as  "range  imaging."  What  is  desired  is  the 
generation  of  accurate,  three-dimensional  coordinate  maps  which  can  be  used 

•  for  environment  definition,  and 

•  to  quantify  processes  to  be  undertaken  or  which  have  just  been 
completed. 

Figure  1  summarizes  the  general  techniques  used  to  generate  a  coordinate  map. 
Independent  of  the  sensing  technology  employed,  range  information  may  be 
acquired  either  through  a  triangulation  procedure,  or  by  measuring  the  time 
for  a  signal  to  propagate  from  a  source  to  a  target  and  back  ("time-of- 
flight").  Each  of  these  techniques  can  again  be  subdivided. 


Figure  1.  Range  imaging  concepts. 

The  triangul ation  technique  can  be  divided  into  passive  and  active  modes. 
In  the  passive  mode,  stereoscopy  is  accomplished  using  two  separate  imaging 
systems  viewing  an  interrogation  volume.  Spatial  coordinates  are  derived  from 
a  triangulation  calculation  which  uses  the  coordinates  of  the  "target"  point 
in  the  image  planes  and  the  known  parameters  of  the  imaging  systems.  In  the 
active  mode,  defined  as  controlled  illumination,  an  imaging  sensor(s)  is  (are) 
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used  to  view  a  volume  which  is  illuminated  by  a  controlled  source.  The 
illumination  may  provide  a  line,  a  point,  or  some  other  combination  which 
either  uses  a  symmetry  of  the  problem  or  is  based  on  a  particular  data  pro¬ 
cessing  scheme.  In  this  approach,  the  known  projection  parameters  of  the 
illuminating  source  are  used  to  constrain  the  problem  and  to  reduce  computa¬ 
tional  complexity. 

Time-of-fl ight  techniques  generally  employ  colinear  sources  and  detec¬ 
tors.  As  with  the  triangulation  approach,  this  technique  can  also  be  divided 
into  two  modes.  In  the  passive  mode,  a  brute  force  approach,  an  impulse 
signal  is  generated  and  the  propagation  time  is  obtained  as  an  elapsed  time 
measurement.  In  the  active  mode,  the  source  is  modulated  in  a  repetitive 
manner  and  the  source  and  reflected  signals  are  compared  to  measure  a  phase 
shift  which  can  be  interpreted  in  terms  of  range. 

These  techniques  are  reviewed  in  the  remainder  of  this  report  and  recom¬ 
mendations  are  made  for  additional  work.  Technologies  specifically  included 
are  optical  stereoscopy,  structured  light,  ultrasonics,  microwaves,  and  laser 
scanners.  A  limited  number  of  miscellaneous  concepts  which  could  eventually 
be  utilized  are  also  included. 
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2.  TRIANGULATION  TECHNIQUES 


To  generate  an  accurate  coordinate  map  with  triangulation  requires  the 
determination  of  two  direction  vectors  to  a  point.  If  these  direction  vectors 
are  separated  by  a  finite  baseline,  then  a  simple  triangulation  calculation 
can  be  used  to  determine  the  intersection  and  spatial  coordinates  of  that 
intersection.  Determination  of  the  needed  direction  vectors  may  be  accom¬ 
plished  using  either  passive  sensor  or  using  one  active  and  one  passive 
sensor. 

The  most  common  form  of  triangulation  is  passive  stereoscopy,  which  uses 
two  passive  imaging  sensors  to  acquire  two-dimensional  images  of  a  volume  of 
interest.  The  sensors  are  usually  optical  imaging  cameras  or  arrays;  direc¬ 
tion  vectors  to  a  specific  point  can  readily  be  generated  by  measuring  the 
coordinates  of  that  point's  image  in  the  image  plane  and  then  using  the  sensor 
optical  parameters  to  calculate  the  direction  vector. 

In  an  alternate  concept,  one  passive  imaging  sensor  is  replaced  by  an 
active  sensor  which  can  interrogate  the  volume  of  Interest  by  a  controlled 
Illumination  source.  In  this  approach,  while  one  direction  vector  is  obtained 
from  the  passive  sensor  as  before,  the  second  is  defined  by  the  illumination 
source.  By  appropriately  coding  and  controlling  this  source,  the  computa¬ 
tional  comp1  tity  of  the  problem  can,  in  many  cases,  be  reduced. 

A  spatial  coordinate  is  generated  from  an  estimate  of  the  intersection 
points  of  a  pair  of  direction  vectors.  The  direction  vectors  are  defined  by 
the  coordinates  of  a  point  in  an  image,  or  by  the  location  of  the  Illumination 
source,  plus  those  system  parameters  which  affect  source  direction  or  image 
signal  direction. 
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2.1  STEREOSCOPY 


The  primary  sensory  system  employed  by  animals,  particularly  humans,  to 
acquire  three-dimensional  information  is  the  eye.  The  human  visual  system 
employs  two  separated  eyes  to  acquire  stereoscopic  imagery  and  exhibits 
excellent  range-finding  and  object  recognition  capabilities.  It  is  only 
natural  then  that  we  attempt  to  mimic  this  ability  in  robotic  systems.  The 
primary  data  acquisition  technology  being  explored  today  for  robotic  systems 
is  optical  stereoscopy. 

Our  familiarity  with  the  general  concept,  plus  the  fact  that  such  a 
system  would  be  passive  (the  only  passive  concept  being  pursued),  make  it 
quite  appealing.  With  stereoscopy,  the  human  eye  is  replaced  by  imaging 
optics  and  by  an  image  sensor.  The  brain's  reasoning  and  computational 
ability  is  replaced  by  hardware  and/or  software.  Acquiring  the  desired 
three-dimensional  information  from  two-dimensional  images  recorded  by  two,  or 
more,  optical  systems  is  conceptually  simple. 

An  outline  of  a  stereoscopic  sensor  system  is  shown  in  Figure  2.  With  a 
knowledge  of  system  parameters,  a  point  can  be  located  in  space  with  a 
knowledge  of  its  coordinates  in  each  image  plane.  Estimating  spatial  location 
is  a  simple  triangulation  calculation  which  can  be  performed  rapidly  and 
accurately. 


fu  J 

|  ^  REAL 

^  WORLD 

u" 

Figure  2.  Stereoscopy  imaging  model. 
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The  stereoscopic  system  may  be  divided  into  three  separate  elements:  the 
optics,  the  image  sensor,  and  the  hardware  and/or  software  required  to  convert 
the  image  data  to  three-dimensional  information.  Current  abilities  in  optics 
will  provide  as  much  resolution  as  Is  needed  for  any  known  or  anticipated 
stereoscopic  system.  The  only  detriment  definable  is  cost.  Commercial 
activities  to  develop  less  expensive  and  simplified  photographic  systems 
represent  the  key  force  in  reducing  this  cost.  Because  of  uncertainties,  the 
direction  vectors  are  more  realistically  conical  and  the  spatial  coordinate 
desired  is  somewhere  within  two  intersecting  cones.  A  least  squares  calcula¬ 
tion  is  generally  performed  to  obtain  most  probable  coordinates. 

As  the  two  sensors  are  separated  to  increase  the  baseline,  the  conical 
error  volume  decreases  until  it  minimizes  at  a  relative  angular  separation  of 
45°.  Beyond  this  point,  the  volume  again  increases. 

All  tri angulation  approaches  have  one  serious  drawback:  with  a  bl static 
system,  both  sensors  may  not  necessarily  be  exposed  to  the  same  regions  of  a 
complex  object.  Obscuration  and  shadowing  may  make  it  impossible  to  develop 
coordinates  for  certain  areas.  Presently,  there  is  no  easy  solution  to  this 
problem.  Attention  has  focused  on  the  Image  sensor  and  also  on  the  hardware/ 
software  required  to  estimate  spatial  coordinates.  Although  not  specifically 
germane  to  this  study,  the  latter  was  included  to  permit  a  better  understand¬ 
ing  of  the  image  sensor  and  its  constraints. 

The  triangulation  computation  Itself  Is  not  difficult  or  time  consuming. 
The  accuracy  of  the  computation  Is  dependent  on  the  accuracy  of  the  optical 
parameters  and  on  the  point  image  coordinates  In  the  Image  plane.  Optical 
parameters,  which  are  usually  known  with  a  high  degree  of  precision,  will 
generally  not  be  a  limiting  factor  in  achieving  excellent  triangulation 
results.  A  key  limitation  Is  the  accuracy  of  the  Image  coordinates  used  In 
the  calculation;  this  accuracy  Is  affected  In  two  ways:  1)  by  the  inherent 
resolution  of  the  Image  sensor,  and  2)  by  the  accuracy  with  which  a  point  can 
be  uniquely  identified  in  the  two  stereoscopic  images.  Ultimately,  the  latter 
constraint  is  the  key  element. 
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Manually,  with  photographically  recorded  images,  coordinate  information 
can  be  acquired  with  a  high  degree  of  precision.  Photographic  film  has  an 
inherently  high  resolution  capability  with  an  image  readily  divisible  into 
millions  of  picture  elements  (pixels).  Additionally,  the  eye  and  brain 
readily  correlate  two  Images  to  identify  a  common  point  with  a  high  degree  of 
accuracy.  It  is  the  latter  which  must  be  efficiently  achieved  in  an  automated 
fashion  to  permit  ready  usage  in  a  robotic  system.  This  difficulty  has  long 
been  recognized  and  considerable  effort  is  being  devoted  to  developing  both 
hardware  and  software  approaches  to  successful  image  correlation.  Grey-scale 
mapping,  edge  enhancement,  vertex  identification,  and  other  techniques  are 
being  explored  to  accomplish  image  correlation.  All  of  these  techniques  are 
dependent  on  the  resolution  ability  and/or  grey-scale  capability  of  the  image 
sensor.  These  capabilities  are  briefly  reviewed  here. 

Both  triangulation  calculations  and  image  correlation  procedures  require 
stable,  wel 1 -regi stered  sets  of  image  data.  Using  camera  systems  with  conven¬ 
tional  photographic  film  as  a  recording  medium  readily  satisfies  this  require¬ 
ment,  but  the  procedures  required  to  generate  useful  data  sets  are  both  labor 
intensive  and  time  consuming.  For  use  in  robotics,  these  operations  must  be 
automated  and  the  image  data  sets  must  be  obtainable  as  direct  analog  or 
digital  electronic  signals. 

The  simplest  electronic  image  sensor  which  can  be  used  for  stereoscopy  is 
the  conventional  television  or  video  camera.  Signal  output,  which  is  analog, 
must  be  converted  to  digital  format  for  computer  usage,  but  it  is  readily 
available  and  relatively  inexpensive.  Video  cameras,  recently  extended  in 
resolution  ability  (up  to  as  high  as  1000  x  1000  pixels),  have  two  distinct 
drawbacks:  1)  they  tend  to  be  relatively  large  in  size,  fragile,  with  signifi¬ 
cantly  high  voltage  requirements;  and  2)  because  of  the  electron  beam  sampling 
used  to  obtain  data,  physical  image  stability  is  not  as  high  as  desired. 

For  these  reasons,  the  image  sensor  of  choice  for  stereoscopy  in  robotics 
is  the  solid-state  imag’  sensor.  Although  there  are  several  competing  tech¬ 
nologies  for  solid  state  image  sensing,  the  most  popular  and  advanced  is  the 
charge-coupled  device  (CCD),  a  silicon  chip  with  a  light-sensitive  surface.  A 
CCD,  which  can  be  manufactured  in  small  size  (postage  stamp  size  is  typical), 


-7- 


GEO-CENTERS,  INC. 


is  a  low-voltage  device.  It  generates  direct  digital  data  and  has  fixed  image 
registration.  A  typical  CCD  consists  of  a  microscopic  grid  of  light-sensitive 
elements  etched  or  deposited  on  a  silicon  chip;  each  element  converts  light, 
striking  it  into  an  electrical  charge.  Pixel  registration  is,  therefore, 
permanent  and  by  careful  mounting  of  a  CCD  pair,  image-to-lmage  registration 
can  be  well  fixed. 

Commercial  imaging  arrays  are  currently  available  at  256  x  256  elements 
(65,000  pixels)  with  8-bit,  grey-scale  ability  (256  levels).  Arrays  having 
double  the  number  of  elements  along  each  axis  (512  x  512)  are  now  available; 
it  is  anticipated  that  1024  x  1024  arrays  will  be  available  In  the  near 
future.  The  largest  single  problem  with  the  current  state-of-the-art  appears 
to  be  picture  element  dropout  and  nonuniformity  of  pixel  response  across  the 
array;  both  are  being  vigorously  addressed. 

The  commercial  video  market  provides  the  impetus  for  technology  develop¬ 
ment.  An  indication  of  the  commercial  applications  of  this  technology  Is 
the  recent  announcement  of  a  magnetic  video  camera  Intended  to  replace  the 
standard  photographic  camera  (Appendix  A).  The  magnetic  video  camera  employs 
a  570  x  490  element  array  and  an  erasable  magnetic  video  disc  intended  for 
playback  and  viewing  with  conventional  color  television  sets.  Such  develop¬ 
ments  represent  key  changes  in  Image  technology.  Only  a  fractional  addition 
to  this  driving  force  is  represented  by  robotics  applications. 

A  sensor  array  may  be  duplicated  in  Imaging  ability  by  scanning  a  linear 
array  across  a  field  of  view,  or  vice  versa.  In  certain  applications,  such  a 
technique  may  be  well-suited  (e.g.,  with  component  motion  on  a  conveyer  belt 
used  to  achieve  scanning).  Linear  arrays  are  readily  available  now  with 
densities  as  high  as  2048  elements.  Scanning  must  be  accomplished  with  array 
motion,  or  with  moving  mirrors,  and  such  designs  lose  generality.  For  this 
reason,  two-dimensional  staring  arrays  are  preferred. 

Although  no  recommendation  Is  being  made  to  support  additional  work  In 
Image  sensors,  or  in  optics  In  general,  it  Is  felt  that  two  areas  are  worth 
consideration.  Both  areas  are  Intended  to  address  the  image  correlation 
problem  and  may  ultimately  Impact  image  sensor  concepts  and  fabrication 
techniques.  In  the  first,  it  Is  felt  that  the  use  of  color  as  a  discriminant 
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should  be  considered  in  developing  image  correlation  techniques.  In  the 
second,  convolution  and  other  computational  approaches  are  being  used  for 
image  correlation  and  it  is  felt  that  the  relatively  new  technology  of  active 
acousto-optic  processing,  and  other  "on-chip"  processing,  may  prove  useful. 

One  of  the  techniques  being  explored  to  achieve  image  correlation  is 
grey-level  matching.  This  approach  may  prove  particularly  useful  in  indus¬ 
trial  applications  where  the  images  considered  have  sharp  discontinuous 
surfaces  emphasized  either  by  shading  or  by  differences  in  angular  reflec¬ 
tivity.  Approaches  being  developed  require  significant  grey-level  discrimina¬ 
tion  and  to  date  have  proven  difficult  to  implement.  Since  the  human  eye 
makes  use  of  color  as  a  discriminant,  it  is  suggested  that  image  correlation 
could  be  advanced  by  a  similar  use  of  color.  Although  effective  correlation 
may  require  all  eight  bits  of  discrimination  in  monochrome  images,  perhaps 
with  an  added  color  discriminant  the  correlation  can  be  accomplished  at  a 
lower  quantization  level.  It  is  suggested  that  a  trade-off  study  may  provide 
interesting  input  to  this  hypothesis. 

Should  a  trade-off  prove  the  utility  of  using  color,  then  imaging  sensor 
technology  would  be  directly  impacted.  Current  technology  produces  a  CCD  with 
a  monochrome  response.  Color  response  is  generated  with  appropriate  sequen¬ 
tial  filtration,  either  by  filtering  three  separate  CCDs,  which  leads  to 
registration  problems,  or  by  sequentially  moving  filters  in  front  of  one  CCD 
(also  not  desirable).  Color  separation  must  be  accomplished  on  one  CCD  with 
adjoining  or  stacked  pixels.  However,  it  is  felt  here  that  the  commercial 
sector  probably  will  provide  the  primary  impetus. 

The  suggested  use  of  active  electro-optic  elements  is  prompted,  first,  by 
the  realization  that  one  approach  to  image  correlation  involves  a  convolution 
operation,  and,  second,  by  the  observation  that  surface-acoustic  waves  can 
readily  accomplish  convolutions  both  rapidly  and  accurately  (Appendix  B). 
Although  convolution  operators  are  being  developed  with  the  understanding  that 
they  will  be  employed  in  a  pipelined  processing  system,  perhaps  they  can  be 
more  efficiently  applied  directly  on  the  CCD  chip.  It  Is  known  that  one 
application  of  this  technology  is  permitting  the  efficient  generation  of 
Fourier  transforms  of  an  image  both  rapidly  and  accurately.  This  technology 
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should  be  explored  in  more  depth,  and  should  appropriate  approaches  be 
developed,  then  Image  sensor  fabrication  will  be  Impacted.  It  Is  not  clear 
that  the  commercial  sector  will  provide  a  significant  driving  force  In  this 
technology,  but  a  decision  to  proceed  can  await  a  successful  demonstration  of 
concept. 

The  most  taxing  application  of  passive  stereoscopy  Is  one  which  has 
images  with  very  slowly  varying  grey-level  content  and  no  discernible  edges  or 
points  which  can  be  used  for  triangulation.  With  such  conditions,  passive 
stereoscopy  may  prove  impossible  or  may  not  be  feasible  without  producing 
significant  errors. 

2.2  CONTROLLED  ILLUMINATION 

Controlled  Illumination  techniques  involve  the  use  of  well-defined  signal 
sources  to  scan  a  volume  of  Interest.  Because  our  ability  to  control  optical 
signals  is  extensive,  and  because  optical  signals  are  minimally  degraded  over 
the  generally  short  ranges  required  for  robotics,  the  preferred  technology  for 
this  application  is  optical.  In  general  terms,  a  light  source  displaced  from 
an  imaging  sensor  is  used  in  a  controlled  Illumination  mode.  Both  the  form  of 
the  source  and  the  manner  In  which  It  is  used  are  controlled  to  maximize  the 
data  acquired  and  to  minimize  computational  complexity.  Typical  light  sources 
for  this  technology  include  light  sheets,  swept  laser  beams,  laser  spots,  and 
other  patterned  formats.  As  opposed  to  the  passive  stereoscopy  described 
previously,  a  range  estimate  Is  simplified  because  the  dimensional  and  angular 
parameters  (direction  vector)  of  the  source  are  well  known. 

A  number  of  systems  employing  some  or  all  of  these  techniques  are  cur¬ 
rently  being  explored  and  developed.  All  have  shown  promise  with  respect  to 
passive  stereoscopy,  but  one  particular  system  appears  to  have  maximum  poten¬ 
tial  (Appendix  C).  In  this  technique,  a  laser  is  used  as  the  illumination 
source  but  its  beam  is  modified  in  a  unique  manner.  Double  interferometry, 
using  two  shearing  plates  at  90°,  is  used  to  generate  a  rectangular  array  of 
controlled  illumination  beams.  This  array  of  beams,  generated  from  one  laser 
source,  exhibits  all  of  the  positive  characteristics  of  laser  illumination  in 
general  and  Is  readily  controlled  as  a  convergent,  divergent,  or  parallel 
array. 
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The  array  is  masked  to  control  the  number  of  elements  (usually  to  a 
symmetrical  array  where  the  number  of  elements  being  used  is  a  multiple  of  2) 
and  to  space-code  the  array  of  spots,  minimizing  the  amount  of  data  needed  to 
uniquely  identify  each  one  imaged.  Images  of  the  space-coded  array  are 
sequentially  obtained,  and  identification  of  a  specific  spot  is  accomplished 
by  simple  image  subtraction.  As  with  passive  stereoscopy,  range  estimates  are 
then  made  by  triangulation  calculations. 
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3.  TIME-OF-FLIGHT  TECHNIQUES 


Direct  ranging  can  be  accomplished  by  means  of  col  inear  sources  and 
detectors  to  directly  measure  the  time  it  takes  a  signal  to  propagate  from 
source  to  target  and  back.  Knowing  the  signal  transport  velocity,  range  is 
then  readily  calculated  from  the  elapsed  transport  time.  The  most  familiar 
use  of  this  technique  is  standard  sonar  technology  in  which  the  echoes  of 
acoustic  pulses  are  recorded  to  provide  reasonable  range  information. 

As  with  the  triangulation  approach,  the  time-of-fl ight  approach  can  be 
accomplished  in  two  ways:  1)  time  of  flight  is  directly  obtained  as  an  elapsed 
time  when  an  impulse  source  Is  used,  and  2)  a  CW  source  signal  is  modulated 
and  the  return  signal  Is  matched  against  the  source  to  measure  phase  differ¬ 
ences.  These  phase  differences  are  then  Interpreted  as  range  measurements. 

Although  optics  (specifically  lasers)  is  again  the  technology  receiving 
the  most  attention,  both  ultrasonics  and  microwaves  have  application.  The 
review  performed  here  centers  on  the  sensing  signal  used  rather  than  the 
technology  employed. 

3.1  LASER  SCANNING 

A  recent  technological  advancement  which  shows  considerable  promise  for 
use  In  robotics  is  laser  scanning.  Lasers  have  been  used  extensively  as 
range-finders,  making  use  of  single  wavelength  operation  and  minimization  of 
beam  divergence.  Simple  ranging  is  accomplished  via  time-of-fl ight  measure¬ 
ment  either  between  a  laser  source  signal  and  a  detector  or  with  signals 
reflected  from  natural  or  man-made  targets. 

Although  originally  developed  as  a  single-point  measurement  technique, 
the  DoD  has  now  pushed  the  technology  Into  an  imaging  mode  which  can  be  used 
for  range  finding,  target  detection  and  identification,  and  moving  target 
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indication.  The  laser  radars  (LIDARS)  developed  for  this  application  are 
sophisticated  units  and  have  abilities  which  are  being  exploited  in  many  new 
weapon  systems.  These  systems,  however,  emphasize  the  longer  range  applica¬ 
tions  needed  for  fielded  military  systems.  For  the  industrial  sector,  shorter 
range  operation  with  even  higher  range  and  angular  resolution  capability  is 
desired.  The  military  laser  scanner  has,  however,  established  feasibility  and 
is  providing  a  technological  base  to  support  the  development  of  robotic 
sensors.  Several  such  systems  have  been  assembled  and  tested  and  development 
work  to  extend  capabilities  is  ongoing  (Appendix  D). 

Conceptually,  the  imaging  laser  scanner  is  well  understood  and  with 
sufficient  care,  assembly  can  be  successful.  Generally,  a  laser  source  is 
used  in  a  pulsed  or  CW  mode  to  illuminate  the  desired  target.  In  the  pulsed 
mode,  time-of-fl ight  range  gating  is  employed;  in  the  CW  mode,  phase  modula¬ 
tion  with  heterodyne  detection  is  used  for  ranging.  In  the  phase  modulated  CW 
mode  of  operation,  an  inherent  range  ambiguity  results.  Care  must  be  taken  to 
ensure  that  inferred  ranges  are  not  in  error  by  the  range  quantum  equivalent 
to  the  modulation  frequency. 

Although  systems  have  been  fabricated  with  range  and  angular  resolution 
capabilities  less  than  1  mm  (at  5-  to  10-foot  ranges),  designers  are  now 
striving  to  achieve  an  order  of  magnitude  improvement.  Figure  3  identifies 
the  major  components  of  such  a  system.  Each  will  be  reviewed  briefly  with 
comments  made  on  those  which  have  potential  for  further  development. 

Operationally,  the  laser  source  must  be  considered  (both  type  and  wave¬ 
length)  as  well  as  the  mode  of  operation  (pulsed  or  CW),  the  beam  scanning 
technology,  and  the  detector  type  to  be  used.  Of  these,  only  the  mode  of 
operation  is  independent,  although  it  is  recognized  that  certain  types  of 
lasers  lend  themselves  more  readily  to  certain  modes  of  operation.  The  CW 
heterodyne  mode  of  operation  is  more  difficult  to  implement  but  is  potentially 
capable  of  greater  range  resolution.  Angular  resolution  is  essentially 
Independent  of  operational  mode  and  is  dependent  only  on  beam  dispersion.  The 
wavelength  of  the  laser  to  be  used  must  be  carefully  selected  to  ensure  1) 
maximum  signal -to-noise  ratios,  and  2)  simplicity  of  operation,  ruggedness, 
and  stability.  It  is  recognized  that  specular  reflections  from  edges  and 
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Figure  3.  Laser  Scanner  System  outline. 


other  discontinuities  can  deterimental ly  affect  data  acquisition.  Such 
reflections  are  readily  observed  at  the  longer  wavelengths  whereas  at  the 
shorter,  ultraviolet  wavelengths  all  surfaces  appear  "rough"  and  specular 
reflections  are  less  likely.  This  represents  one  definite  advantage  to  using 
shorter  wavelength  sources. 

There  is  a  trend  toward  more  compact,  lower-cost  laser  systems.  It  is 
felt,  for  example,  that  6-inch  long  helium-neon  lasers  will  shortly  be  avail¬ 
able.  These  small,  stable,  multimode  lasers  will  have  increased  use  in 
battery-powered  portable  scanning  units.  There  are  comparable  advancements 
being  made  in  semiconductor  laser  technology.  These  are  of  special  interest 
for  the  application  of  laser  scanning  systems  to  robotics. 

Standard  techniques  for  beam  scanning  use  moving  mirrors  and  rotating 
prisms.  Several  new  technologies  have  recently  shown  increased  promise. 
These  include  holographic  and  acousto-optic  techniques.  Neither  shows  any 
current  advantage  over  more  standard  techniques  for  robotics  application. 

The  development  of  holographic  scanners  has  been  a  significant  recent 
advancement.  This  technology  has  been  advanced  by  the  commercial  sector, 
primarily  for  data  acquisition  systems  such  as  point-of-sale  product  code 
scanning.  In  this  application,  a  spinning  disc,  containing  a  number  of 
transmission  holograms,  is  used  to  deflect  and  focus  a  laser  beam  by  diffrac¬ 
tion.  Efficiencies  of  these  holographic  scanners  have  exceeded  90%,  and  show 
considerable  promise  for  replacing  rotating  polygon  spinners  for  the  same 
purpose. 

Rotating  polygon  spinners  are  also  used  for  beam  scanning  but  must  be 
manufactured  to  extremely  tight  tolerances.  Typical  requirements  are  frac¬ 
tional  wavelength  flatness  per  surface  and  a  surface-to-surface  orientation 
tolerance  of  less  than  several  arc  seconds.  New  techniques  in  diamond  point 
machining  and  on-line  measurements  are  making  these  objectives  attainable,  but 
polygon  elements  are  still  extremely  expensive.  Holographic  scanners,  on  the 
other  hand,  could  be  replicated  very  inexpensively  by  holographic  recording 
techniques. 
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Acousto-optic  beam  scanning  capitalizes  on  the  fact  that  the  index  of 
refraction  can  be  changed  by  applying  pressure  to  the  crystal.  In  Figure  4, 
the  entering  laser  beam  will  be  diffracted  by  the  crystal.  As  pressure  is 
applied  to  the  crystal,  its  refractive  index  will  change  and  the  laser  beam 
deflection  will  be  modified  accordingly.  In  practice,  pressure  is  applied  to 
the  crystal  through  a  piezo-electric  material.  By  modulating  the  driving 
signal  sent  to  the  piezo-electric  material,  the  laser  beam  is  deflected. 
Although  advances  in  this  technology  have  been  dramatic  and  useful,  acousto¬ 
optic  modulation  for  beam  scanning  is  still  limited  to  small  fractions  of  a 
degree.  The  applications  envisioned  here  would  require  tens  of  degrees  of 
deflection,  while  still  maintaining  beam  integrity. 
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Figure  4.  Outline  of  Acousto-Optic  Laser  Beam  Diffraction. 

Scanning  with  moving  mirrors  (galvanometers  or  resonant  scanners)  remains 
one  of  the  easiest  technologies  to  Implement  and  is  the  least  expensive. 
However,  significant  limitations  are  placed  on  the  performance  of  such  systems 
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by  the  inertial  mass  of  the  oscillating  mirror.  Current  technology  allows 
operation  up  to  ~500  Hz,  but  advances  in  new  lightweight  substrate  materials 
will  allow  operation  at  higher  limits.  The  technology  is  still  fragile, 
however,  and  not  the  most  desirable. 

3.2  ULTRASONICS 

Active  ultrasonic  interrogation  is  regularly  used  to  acquire  accurate 
ranging  information.  The  ultrasonic  range  finders  used  on  some  of  the  newer 
camera  systems,  for  example,  are  capable  of  0.1-foot  resolution  in  the  range 
of  5  to  35  feet.  However,  the  beam  width  of  the  emitting  source  is  almost  a 
full  20°;  therefore,  the  system  angular  resolution  is  limited. 

The  use  of  ultrasonics  for  imaging  or  range-finding  has  two  inherent 
limitations.  First,  ultrasonic  signals  are  severely  attenuated  in  air  with 
1  dB/m  being  readily  observed.  In  a  fluid  medium,  attenuation  is  not  as 
severe.  Sonar  systems  are  regularly  employed  in  underwater  applications  and 
are  even  used  ’  n  internal  medical  imaging  applications.  As  a  result  of 
attenuation  limitations,  it  is  difficult  to  define  an  ultrasonic  imaging 
system  with  an  appreciable  range.  Secondly,  propagation  of  ultrasonic  signals 
is  a  physical  molecule-to-molecule  or  atom-to-atom  process.  As  such,  the 
random  thermal  motion  of  atmospheric  species  is  superimposed  on  the  direction 
of  propagation.  This  assures  a  significant  beam  spread  with  attendant  loss  of 
resolution.  Also  implied  here  is  a  significant  problem  with  respect  to 
temperature  dependence. 

For  certain  applications,  ultrasonics  may  prove  to  be  the  technology  of 
choice.  The  versatility,  cost,  speed,  and  accuracy  of  short  range  systems  are 
highly  desirable  and  should  be  explored  for  applications  such  as  proximity 
sensing  and  high  accuracy  parts  or  systems  inspection.  A  unique  application 
of  ultrasonics,  phase  monitoring,  has  recently  been  developed  and  shows  great 
promise  for  specific  applications  (Appendix  E). 

With  phase  monitoring  (PM),  an  ultrasonic  source  Is  directed  at  the 
object  or  system  to  be  considered.  The  sound  waves  constructively  and  de¬ 
structively  Jnterfere  to  produce  a  standing  wave  pattern  which  is  sampled  by 
an  array  of  detectors,  usually  sample  microphones.  With  a  relatively  small 
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computational  and  data  storage  ability,  the  PM  system  can  be  "trained"  to 
recognize  the  pattern  created  by  a  finite  data  set.  This  ability  can  then  be 
used  for  high  tolerance  automated  inspection  and  for  limited  command  ability. 

One  positive  aspect  of  using  the  PM  technique  is  that  it  has  the  limited 
ability  to  “see  around  corners."  With  optical  sensing  techniques,  data  are 
acquired  only  on  surfaces  that  can  be  viewed  directly.  With  PM,  the  standing 
wave  pattern  will  be  affected  by  contributions  from  all  sources.  This 
includes  reflections  (even  though  multiple)  from  surfaces  not  directly  seen  by 
the  source. 

For  automated  inspection,  PM  has  proven  to  be  extremely  valuable,  rapid, 
and  accurate.  An  object  placed  in  the  acoustic  field  of  an  ultrasonic  source 
will  uniquely  perturb  the  field.  By  sampling  the  field  at  a  number  of  loca¬ 
tions  with  an  array  of  detectors,  field  deviations  created  by  small  object 
changes  can  readily  be  detected. 

The  general  concept  is  illustrated  in  Figure  5.  The  source  first  illumi¬ 
nates  a  calibration  object;  the  resultant  standing  field  is  sampled  by  a 
microphone  array.  The  "standard"  field  pattern  is  stored  in  memory  and  the 
measured  patterns  for  objects  to  be  tested  or  inspected  are  matched  against 
the  standard.  Both  displacement  and  surface  defect  perturbations  are  detect¬ 
able.  Position  errors  as  small  as  1  mil  (0.03  mm)  and  defect  volumes  as  small 
as  0.002  in. 3  (30  mm3)  have  been  detected  at  frequencies  of  10  to  20  kHz. 

Such  sensitivity  is  well  demonstrated  by  the  fact  that  such  a  system  can 
differentiate  between  heads  or  tails  on  a  coin.  Note  however,  that  if  the 
coin  is  not  introduced  with  a  consistent  orientation  (e.g.,  head  always 
pointing  in  the  same  direction),  the  system  loses  the  ability  to  uniquely 
identify  status.  A  limited  ability  to  accommodate  rotation  can  be  acquired  by 
expanding  the  training  set  data  base,  and/or  exploiting  the  rotational 
symmetry  of  the  problem  in  hardware  or  software. 

A  PM  system  also  has  a  limited  ability  to  compensate  for  slight  errors  in 
test  object  placement.  The  source/microphone  array  (the  relative  location  of 
source  and  microphones  must  be  held  fixed)  may  be  moved  and/or  the  object  may 
be  moved  in  an  attempt  to  improve  the  pattern  match  obtained.  Both  theory  and 
experiment  have  demonstrated,  however,  that  if  the  initial  placement  is  not 
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close  to  that  sought,  movement  instructions  based  on  sampled  data  may  actually 
result  in  a  divergence  from  the  desired  position.  To  prevent  this  from 
happening,  initial  placement  should  not  exceed  half  a  wavelength  from  the 
"standard"  (~0.5  inch  at  10  kHz). 


ACOUSTIC 


Figure  5.  Outline  of  Ultrasonic  Phase  Monitoring  Technique. 
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4.  CONCLUSIONS  AND  RECOMMENDATIONS 


The  ability  to  acquire  spatial  information  for  robotic  applications  has 
improved  considerably  in  the  last  several  years.  Improvements  have  resulted 
from  the  utilization  of  new  technologies  and  from  advancements  in  the  applica¬ 
tion  of  older  technologies.  It  is  certain  that  this  growth  will  continue  and 
that  commercial  applications  will  provide  a  significant  impetus  to  this 
growth.  During  this  review,  several  technological  areas  were  identified  which 
are  key  to  a  continued  growth  over  the  long  term.  The  following  areas  are 
specifically  recommended  for  additional  research: 

•  Image  Correlation  Techniques  —  needed  to  ensure  that  stereoscopy  can 
be  used  in  a  timely  and  efficient  manner.  Recommended  specifically  is  an 
examination  of  the  use  of  color  as  a  discriminant  for  correlation  algorithms. 
Grey  scale  has  been  used  extensively,  but  it  is  felt  that  the  added  use  of 
color  would  simplify  correlation  algorithms  and  require  fewer  digitization 
levels. 

Independent  of  the  image  correlation  technique(s)  ultimately  used  for 
stereoscopy,  efforts  should  be  undertaken  to  shift  the  image  processing  from 
the  software  world  where  It  is  usually  developed  to  an  implementation  in 
hardware.  This  would  minimize  the  amount  of  digital  information  which  must  be 
manipulated  and  would  also  significantly  enhance  data  processing  rates.  A 
study  of  generic  image  processing  techniques  should  be  undertaken  to  determine 
which  are  capable  of  formulation  as  "on-chip"  processing  elements.  Key  to 
such  a  study  would  be  determination  of  the  amount  of  data  which  must  be  passed 
from  pixel  to  pixel  in  an  image  as  well  as  between  images. 

•  Laser  Technology  —  represents  one  of  the  keys  to  several  of  the  data 
acquisition  techniques  reviewed  in  this  study.  To  enhance  this  technology, 
two  areas  must  be  addressed: 
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1)  Small  high-power  semiconductor  lasers  should  be  developed  (preferably 
without  cooling  requirements).  This  permits  the  mounting  of  an 
active  laser  probe  at  positions  of  optimum  use.  It  also  minimizes 
the  use  of  sophisticated  beam  transmission  techniques  which  increase 
computational  complexity  and  are  difficult  to  maintain. 

2)  Nonmechanical  beam  deflection  techniques  need  to  be  developed.  The 
rotating  or  oscillating  techniques  currently  used  are  not  rugged  and 
require  considerable  skill  and  competence  to  maintain  alignment. 
Desired  here  would  be  the  development  of  techniques  similar  to  the 
acousto-optic  deflectors  currently  used  for  laser  printing  and 
optical  character  recognition  schemes.  These  systems  are  only 
capable  of  total  laser  beam  deflections  on  the  order  of  a  fraction  of 
a  degree.  For  use  in  robotic  applications,  deflections  on  the  order 
of  tens  of  degrees  are  required. 

•  Proximity  Sensing  —  needs  development  as  a  complement  to  the  data 
acquisition  techniques  reviewed.  It  Is  recognized  that  all  of  these  techni¬ 
ques  have  limited  resolution  abilities  and  that  all  will  eventually  be  detri¬ 
mentally  affected  by  manipulators  or  other  hardware  as  a  close  approach  is 
attempted.  Both  Fiber  Optics  Systems,  with  a  transmitted  light  beam,  and 
Ultrasonics  should  be  examined  for  this  application.  Both  have  shown  promise 
and  both  may  eventually  be  useful  for  specific  applications. 

In  summary,  recommendations  are  made  that  additional  efforts  be  under¬ 
taken  in: 

•  Image  Correlation  Techniques 

-  Color  as  a  discriminant 

-  "On-chip"  processing 

•  Laser  Technology 

-  Development  of  higher  power  semiconductor  sources 

-  Development  of  nonmechanical  scanning  techniques 

•  Proximity  Sensing 

-  Fiber  Optics 

-  Ultrasonics 
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APPENDIX  A 

SOLID  STATE  IMAGING  TECHNOLOGY 


The  development  of  high-speed,  accurate  triangulation  techniques  for 
acquiring  range  information  requires  high  resolution,  solid  state  image 
sensors.  The  development  of  this  technology  has  been  rapid,  but  the  major 
driving  force  for  future  progress  will  come  from  the  commercial  sector. 
Originally,  solid  state  imaging  concepts  were  explored  for  applications  in 
space  systems  and  in  weapons  or  weapons  delivery  systems.  Both  applications 
require  compact,  lightweight,  rugged  sensors. 

The  commercial  sector  is  now  the  major  user  of  the  technology  and  the 
attached  article  describing  a  commercial  application  supports  this  view.  The 
capabilities  reviewed  here  are  impressive  and  there  are  indications  that 
improvements  can  be  expected.  Effective  commercialization  of  the  concept 
described  in  this  Appendix  for  a  mass  market  requires  high-volume,  low-cost 
production.  These  benefits  are  of  interest  for  robotics  applications. 
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Photography 
Joins  the  Electronic  Age 


JON  WEINER 


New  low-cost  computers  have  con¬ 
vened  the  filing  process  into  a 
disappearing  act  for  documents — that 
is,  all  documents  except  photographs. 
Data  can  be  bandied  wuhoui  recourse 
to  paper,  but  although  the  technology 
for  converting  images  into  digital  sig¬ 
nals  has  been  developed  (witness  the 
dramatic  photographs  of  the  outer 
planets),  most  companies  cannot  |ue 
tify  the  expense  for  routine  filing 
They  therefore  resort  to  the  time-hon¬ 
ored  method  of  retrieving  dog-eared 
phorographs  from  manila  folders 
Not  for  long  In  the  fall  of  1961  the 
Sony  Corporation  unveiled  in  the 
United  States  the  prototype  of  a  "film¬ 
less  camera"  that  substitutes  sophist  1 
cited  electronics  for  film  and  uses  a 
television  screen  instead  of  coated  pa 
per  to  display  each  color  still  shot 
When  the  Mavica— short  for  magnetic 
video  camera — is  ready  for  distribu¬ 
tion  in  1983,  ii  will  no  doubt  be  chal¬ 
lenged  by  a  host  of  Sony's  compel! 
tors,  who,  despite  disclaimers,  are 
working  on  similar  products. 

Sony  chairman  Aluo  Moriu  called 
the  Mavica  "a  revolution  in  photo¬ 
graphic  history,"  and  the  world  press 
played  it  up  as  the  fust  giant  step  in 
photography  since  Daguerre  invented 
the  first  practical  photographic  pro¬ 
cess  140  years  ago.  But  the  hlmless 
camera  is  really  only  an  extension  of 
video  technology.  No  one  expects  it  to 
replace  the  35mm  camera,  much  less 
drive  Polaroid  out  of  business,  but  the 
video  camera  will  certainly  find  a 
ready  market  with  upscale  consumers 
who  enioy  taking  family  snapshots 
That  a  picture  can  be  immediately 
seen  on  any  color  television — a  sort  of 
instant  TV  Polaroid — should  be  enor 
roously  appealing  to  amateur  shutter- 
bugs  The  Mavica  is  also  likely  to  find 
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A  char  ft  -coupled  device  (CCD)  is  the 
been  of  filmiess  video  camtias. 

a  significant  market  in  businesses 
that  must  keep  a  large  number  of  pic¬ 
tures  on  hie  'It  will  be  good  for  news 
applications  and  other  specialized  ap 
plications,  not  for  appreciating  a  great 
subfect  with  depth,"  explains  James 
Chung,  who  follows  the  photography 
market  for  Merrill  Lynch 
Like  the  Walkman  portable  stereo 
cassette  player  or  the  Tummy  TV,  the 
Mavica  bears  Sony's  trademark  of 
practicality  and  convenience,  com¬ 
bined  in  a  package  so  miniature  it  in¬ 
vites  use.  It  resembles  a  conventional 
single-lens  reflex  (SLR)  camera,  al¬ 
though  it  is  a  bit  heavier  at  800  grams 
Like  most  SLR's,  the  Mavica  has  inter¬ 
changeable  lenses  (so  far,  Sony  plans  a 
25mm  F2,  a  50mm  F  1.4  and  a  4  tunes 
zoom  F  1.4  from  1 6mm  to  64mm)  and 
a  hinged  minor  to  permit  through-the 
lens  viewing.  It  can  shoot  single 
frames  at  shutter  speeds  from  1/60  to 
1  /1 ,000  second  or  make  continuous  re¬ 
cordings  of  up  to  10  pictures  per  sec¬ 
ond.  It  shoots  color  pictures  ar  ASA 
200,  about  the  speed  of  fast  color  films 
Instead  of  a  roll  of  him,  however, 
the  Mavica  uses  a  6  by  .03-centimeter 
floppy  magnetic  disc  called  the  Mavi- 


t  pak,  which  stores  uj  to  50  color  pic¬ 
s'  tures  Essentially  a  small  videodisc,  it 
1  can  be  inserted  in  a  special  viewer  for 
::  displaying  the  images  on  an  ordinary 
*  television  screen 

The  disc  can  be  taken  out  of  the 
camera  and  viewed  after  only  a  few 
pictures  are  shot  and  then  returned  to 
the  camera  It  can  be  erased  and  used 
over  and  over  again,  like  video  tape 
And  individual  frames  can  easily  be 
transferred  onto  video  tape  to  make  a 
video  album.  Sony  has  plans  for  a  pic¬ 
ture  printer  that  will  make  color 
prints  (five  by  seven  inches  or  smaller) 
Monta  estimates  the  camera's  retail 
price  will  be  S650,  plus  about  $220  foi 
the  TV-display  viewer  and  at  least 
$200  for  the  hard-copy  printer.  Thus 
the  system  will  probably  cost  just 
over  $1,000  when  ii  fust  enters  the 
market.  Only  the  "film"  is  cheap:  the 
reusable  magnetic  disc,  in  a  hard  plas¬ 
tic  case,  will  cost  $2.50 
The  Mavica  can  be  made  so  mm  be¬ 
cause  Sony  replaced  the  conventional 
vidicon  tube,  which  is  heavy  and  frag¬ 
ile,  with  a  silicon  chip  that  has  a  lighi- 
sensitive  surface  This  remarkable 
new  image  sensor — called  a  charge- 
coupled  device  (CCD) — is  about  the 
size  of  a  postage  stamp,  but  it  ac¬ 
counts  for  a  considerable  part  of  the 
Mavica'spnce. 

CCD's  were  invented  by  Willard 
Boyle  and  George  Smith  at  Bell  Labo¬ 
ratories  in  1969  Some  black-and- 
white  miniature  video  cameras  were 
made  with  CCD's  as  early  as  197J 
Sony  has  not  revealed  bow  it  produces 
a  color  picture  using  CCD's  A  CCD  is 
a  microscopic  gnd  made  up  of  light- 
sensitive  squares,  each  of  which  con 
verts  the  light  that  strikes  it  into  an 
electric  charge.  Each  of  the  squares 
represents  one  bit  of  information — a 
pixel,  or  picture  element— and  is  ap 
proximately  the  sue  of  tbc  black  dots 
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that  make  up  newspaper  pictures. 

To  transfer  all  these  pixels  into  the 
memory  of  the  magnetic  disc,  the 
CCD  uses  an  electnc  field  to  pass 
charges  to  the  edge  of  the  gnd  At  the 
moment  each  charge  reaches  the  edge 
■t  is  measured,  and  the  information  is 
stored  in  the  video  disc. 

At  present  a  picture  taken  with  the 
Mavica  is  slightly  fuzzy  because  the 
CCD  contains  only  570  horizontal 
imaging  elements  and  490  vertical 
imaging  elements — fewer  than 
280,000  pixels  m  all.  Morita  says  that 
the  resolution  will  improve  jand  ex¬ 
pects  costs  to  come  down),  but  it  may 
be  many  years  before  the  CCD  can 
match  the  high  quality  of  35mm 
films,  whose  fine  grain  is  the  equiv¬ 
alent  of  one  million  pixels  per  picture. 

The  beauty  of  the  Mavica 's  magnet  - 
ic  memory  is  that  information  from  it 
can  be  converted  instantly  into  a  digi¬ 
tal  signal  and  transmitted  quickly  and 
simply  over  telephone  wires  A  pho¬ 
tographer  halfway  around  the  world 
could  put  the  disc  into  a  transmitter 
that  digitizes  the  signals,  and  off  the 
images  would  go  to  the  home  office. 
Of  course,  Wi  rephoto  is  nothing  new 
to  AP  and  UP1,  but  the  wire  services 
are  currently  forced  to  rely  on  him 
that  is  processed  and  printed  on-site 
and  on  expensive,  elaborate  scanning 


systems  that  convert  the  images  into 
electronic  signals. 

F.  W.  Lyon,  vice-president  for  news 
pictures  at  UP1,  is  "very  interested"  in 
the  Mavica,  but  be  has  challenged 
Sony  to  improve  its  resolution  to  that 
of  35mm  film.  On  the  other  hand.  Bob 
Cerson,  senior  editor  of  Television  Di¬ 
gest,  says  that  if  it  were  possible 
to  use  some  of  the  image-enhance¬ 
ment  techniques  developed  by  NASA, 
which  blend  scan  lines  into  a  continu¬ 
ous  image,  then  "in  theory,  this  could 
give  a  hard  pnnt  from  the  Mavica  a  lot 
more  quality.  Not  great— but  you're 
only  talking  about  a  three-by  hve 
inch  pnnt." 

Filmless  cameras  will  have  other 
specialized  applications,  according  to 
Harry  Machida,  manager  for  financial 
corporate  communications  at  Sony. 
Insurance  companies  requue  millions 
of  low-quality  photographs  fot  their 
records,  and  photographs  taken  by  the 
video  camera  will  be  easy  to  hie,  store 
and  retrieve  electronically.  For  the 
same  reason,  the  military  and  police 
will  find  the  video  system  attractive. 

Because  the  Mavica  can  be  connect¬ 
ed  with  a  special  adapter  directly  to  a 
borne  video  tape  recorder  such  as 
Sony's  Betamax,  it  can  be  used  as  a 
live  video  camera.  There  are  currently 
3  million  video  cassette  recorders  in 


the  United  Sutes,  and  no  one  expects 
the  recent  copyright  ruling,  which  re 
stricted  video  taping,  to  dampen  sales 
One  out  of  every  hve  VCR  owners  also 
buys  a  conventional  portable  video 
camera,  which  costs  anywhere  from 
$500  to  $1,400  Against  those  prices 
the  Mavica  is  already  competitive. 

Sony  faces  stiff  competition  in  the 
future— and  may  not  be  far  ahead  of 
the  pack.  Sharp  Corporation  of  fapan 
has  announced  that  it  is  preparing  to 
market  a  similar  camera  that  weighs 
270  grams  less,-  Sharp  will  distnbute  it 
in  fapan  in  the  fall  of  1982.  Several 
other  fapanese  electronics  firms  and 
American  companies  such  as  Texas 
Instruments,  RCA,  Kodak  and  Polar 
oid  are  rumored  to  be  working  on 
similar  systems. 

But  this  kind  of  competition  does 
not  worry  Sony  overmuch.  A  report  by 
securities  analyst  Brenda  Landry  of 
the  investment  firm  Morgan  Stanley 
notes  that  "the  company  would  prefer 
to  position  itself  in  a  business  with 
good  growth  potential  even  though 
there  may  be  compel  itots  rather  than 
have  a  slow-growing  area  all  to  itself  " 
And  the  growth  potential  is  unques 
tionable.  Says  Katherine  Stults  of 
Morgan  Stanley,  "The  video  revolu¬ 
tion  is  real  And  the  Mavica  becomes 
one  more  piece  in  t’  atfcle."  □ 


THE  RIMLESS  CAMERA 


Sony’s  magnetic  video  camera  uses  a  tiny  CCD  image  sensor  to  convert  light  directly  into  electric  signals  The  signals  are  stored  on  a 
floppy  magnetic  disc  called  the  Mavipak,  which  can  store  up  io  SO  color  still  pictures.  The  video  disc  is  inserted  into  a  special  viewer 
in  order  io  display  the  pictures  on  an  ordinary  television  set.  The  disc  can  be  erased  and  used  over  and  over. 
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APPENDIX  B 

ON-CHIP  IMAGE  PROCESSING 


Automated  stereoscopy  requires  the  development  of  high-speed,  efficient 
techniques  to  correlate  the  two  images  to  be  used.  Many  of  the  concepts  being 
explored  to  accomplish  the  desired  image  correlation  require  extensive  compu¬ 
tational  effort.  However,  some  of  these  mathematical  processes  are  amenable 
to  execution  with  hardware  as  opposed  to  software. 

Technological  advances  in  both  active  processing  elements  and  in  higher 
density  computational  elements  may  be  capable  of  implementation  directly  on  an 
image  sensor  chip.  Packing  densities  for  computational  elements  have  steadily 
been  increasing  and  have  resulted  in  smaller,  higher  speed  modules.  Addition¬ 
ally,  active  processing  has  been  developed  which  exhibits  capabilities  of 
direct  Interest  to  imaging  for  robotics. 

A  technique  which  can  be  used  to  acquire  Fourier  transforms  of  images  on 
a  real-time  basis  is  presented  on  the  following  pages.*  The  use  of  surface 
acoustic  wave  technology  to  perform  the  bulk  of  the  processing  required  is 
unique  and  dramatically  reduces  the  amount  of  numerical  processing  required. 
Such  techniques,  or  combinations  thereof,  should  be  explored  more  fully  for 
this  application. 


♦Reprinted  from  the  July  1980  issue  of  Optical  Spectra. 
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OPTICAMMAGERS:  THE  DEFT  CAMERA 


By  Stephen  T.  Kowel 

The  relationship  between  optical  Im¬ 
agers.  such  as  the  CCD  errey.  and  opti¬ 
cal  Fourier  transformers  is  similar  to 
the  one  between  oscilloscopes  and 
spectrum  analyzers.  While  optical  im¬ 
agers  make  plots  of  image  intensity  as 
a  function  of  position.  Fourier  trans¬ 
formers  look  for  the  spetial  frequency 
content  of  optical  images.  Bee e use  of 
this  difference.  Fourier  transformers 
are  better  suited  for  processing  appli¬ 
cations.  including  image  alignment, 
focus  detection  and  motion  detection, 
than  standard  optical  imagers. 

The  direct  electronic  Fourier  trans¬ 
form  (DEFT)  sensor  takes  full  advan¬ 
tage  of  Fourier  imeging.  It  can  elec¬ 
tronically  select  erbitrary.  two-dimen¬ 
sional.  Fourier  components  of  arbi¬ 
trary  images  through  e  novel  pseudo¬ 
beam  steering  technique 

DEFT  structure 

The  DEFT  camera  (Figure  1)  con¬ 
sists  essentially  of  e  photoconducting 
film  of  cadmium  sulfide  (CdS)  depos¬ 
ited  on  a  piezoelectric  subetrete  (liN- 
bO,)  A  suitable  metal  pattern  is  evap¬ 
orated  onto  the  CdS  to  pick  up  photo¬ 
currant.  Interdigital  transducers  era 
used  to  generete  two  orthogonal  sur¬ 
face  acoustic  waves  in  the  substrate. 

In  operation,  the  DEFT  camera  fo¬ 
cuses  the  optical  image  in  its  field  onto 
the  CdS  film.  The  electric  fields  associ¬ 
ated  with  the  acoustic  waves  induce  a 
nonlinear  modulation  of  the  conductiv¬ 
ity. 

A  full  tensor  treatment  of  this  inter¬ 
action'  reveals  that  the  deposited  con¬ 
tacts  detect  a  currant  proportional  to 
l(t) «  exp  IHw,  -  w,)t)  /  d*T  I(r  )exp 
M*T) 

where  1(T )  is  the  image  intensity,  w, 
and  oj,  are  the  frequencies  of  the  two 
acoustic  waves  and  l  has  as  its  com¬ 
ponents  the  wave  vectors  of  the  two 
acoustic  waves.  By  varying  the  acous¬ 
tic  frequencies,  we  can  vary  land 
probe  different  points  in  the  Fourier 
apece.  Under  these  conditions,  the 
signal  behaves  as  If  a  new  acoustic 
wave  has  been  created  with  e  wave 
vector  equal  to  the  sum  of  the  acoustic 
wave  vectors.  We  call  this  effect  pseu¬ 
do-beam  steering 


Unlike  digital  techniques  of  Fourier 
transformation,  which  digitize  image 
information  after  suitable  image  scan¬ 
ning.  the  analog  DEFT  technique  ex¬ 
ploits  physical  material  properties  to 
extract  Fourier  information  in  real 
time.  There  is  no  need  for  digitization. 

Sensing  plus  preprocessing 

In  a  number  of  image  processing  ap¬ 
plications.  the  unique  preprocessing 
capability  of  the  DEFT  sensor  offers 
advantages  over  alternate  methods  of 
image  sensing,  such  as  raster  scanning 
or  optical  Fourier  transformation.1  Im¬ 
age  sensing  and  preprocessing  are 


combined  in  e  single  device  without 
the  need  for  e  coherent  light  source, 
expensive  optical  components  or  pre¬ 
cision  alignment. 

The  utility  of  the  DEFT  technique 
can  be  illustrated  by  its  application  to 
some  typical  imege  processing  func¬ 
tions.  Hera  we  outline  four  such  exam¬ 
ples:  image  alignment,  focus  detection, 
motion  detection  and  pattern  recogni¬ 
tion. 

Recently,  Deft  Laboratories  built  an 
experimental  system  for  automatic  im¬ 
age  alignment  with  respect  to  a  refer¬ 
ence  image  using  DEFT  sensors.  In  the 
system,  two  matched  DEFT  sensors 


B-2 


look  at  two  identical  but  misaligned  im¬ 
ages  and  provide  Fourier  components 
to  e  microcomputer.  The  microcompu¬ 
ter  determines  misalignment  in  x.  y 
and  6,  using  a  special  algorithm  based 
on  the  Fourier  transform  spece-shift- 
in g  theorem.  Since  the  transform  mag¬ 
nitude  functions  ere  invariant  to  trans¬ 
lational  misalignment,  hy  computing 
their  cross-correlation  as  a  function  of 
angle  you  can  determine  the  angle  mis 
alignment.  A6  The  misaligned  image  is 
than  rotated  to  remove  AS. 

Once  A8  is  removed.  m^Ax  4  u>yAy  is 
determined  at  e  number  of  spatial  fre¬ 


quency  samples  using  the  space-shift¬ 
ing  theorem  relationship.  Then  Ax  and 
Ay  are  determined  using  leest  squares 
estimation. 

The  algorithm  is  complicated  by  the 
fact  thet  true  phese  values  are  re¬ 
quired  but  only  principal  values  of 
phase  ere  available.  This  leads  to  an 
iterative  process  wherehy  increasing¬ 
ly  higher  spetial  frequencies  are  used 
to  give  increasingly  better  estimates  of 
Ax  and  Ay.  The  system  is  described  in 
greater  detail  in  reference  3. 

The  number  of  operations  (multipli¬ 
cations  or  divisions)  required  to  align 


two  images  using  the  algorithm  is  ap 
proximetely 

n,(2n,  4  2)  4  n,,(5nf  4  12)ops  (2| 

where  nt  is  the  number  of  sample  val¬ 
ues  used  to  represent  the  image  or 
transform .  n,  is  the  number  of  angle  in 
crements  used  during  correlation  and 
nl,«n4  is  the  number  of  least-squares 
estimates  In  contrast,  alignment  based 
on  correlation  of  the  image  intensity 
functions  requires 

n,n>nJ2n>  4  2)ops  (3) 

where  nt  and  ny  are  the  number  of  x 
and  y  increments.  Because  the  DEFT 
sensors  provide  Fourier  transforms, 
images  ere  not  used  directly  and  a  one 
dimensional  cross-correlation  plus  e 
leest-equeres  estimation  of  two  para¬ 
meters  replaces  e  less  efficient  three- 
dimensionel  cross-correletion 

The  present  alignment  system  cen 
align  e  25-centimeter-square  image  to 
within  50  micrometers  in  x  and  y  coor¬ 
dinates  and  to  0.1  degree  in  6.  The  next 
generation  of  DEFT  cameras  will  have 
better  signal-to-noise  retio  and  lerger 
bandwidth  for  significantly  better 
overall  performance 

Automatic  locus  detection 

Focus  detection  with  the  DEFT  sen¬ 
sor  relies  on  the  feet  thet  most  focused 

Ftflur*  2. 

1M  OUECl  SHOWN  M  A  WAS  IMAGED  WITH  A  DEO 

HNSoe  s  shows  SfNsoe  ouirur  whin  the  image 

Of  ABMfOCEJS  C  SHOWS  MNSOCOUTfVTFO*  AN 
OOT-OWOCOS  MAGE  Of  A 
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images  contain  abrupt  changes  in  in 
tensity  due  toohjert  edges.  As  a  result, 
a  focused  image  contains  greater  high 
spatial  frequency  content,  which  can 
be  detected  by  the  DEFT  sensor. 

You  can  visualise  the  sensor's  po¬ 
tential  for  automatic  focus  detection 
by  a  simple  experiment  The  object  of 
Figure  2A  was  imaged  on  a  DEFT  sen¬ 
sor  Figures  2B  and  2C  are  plots  of  ac¬ 
tual  sensor  output.  The  axes  are  those 
in  two-dimensional  Fourier  space,  and 
the  peaks  indicate  the  prominent  spa¬ 
tial  frequencies  contained  in  the  image 
of  Figure  2A. 

When  the  image  is  in  focus,  the  sen¬ 
sor  produces  the  output  shown  in 
Figure  2B.  The  largest  peak  is  the 
taro  spetial  frequency  component  The 
other  two  peaks  are  the  first  and  third 
harmonica  of  the  spatial  periodicity  in 
the  image  Figure  2C  shows  the  sensor 
output  with  the  image  slightly  out  of 
focus.  Notice  that  the  first  harmonic  is 
smaller  and  the  third  harmonic  has  vir¬ 
tually  disappeared 

An  automatic  focus  detection  sys¬ 
tem  using  the  DEFT  sensor  would  op¬ 
erate  as  follows:  The  sensor  views  the 
image  to  be  focused  and  the  lens  is 
moved  through  its  extremes  of  travel 
At  a  sufficiently  high  sampling  rate, 
the  sensor  is  addressed  at  one  or  more 
spatial  frequencies  For  example,  let 
the  lens  be  at  position  xt  at  sample  1 . 
At  each  xt.  the  magnitudes  of  the  ad¬ 
dressed  Fourier  components  are  read 
and  averaged  to  provide  a  signal  to  be 
maximized  Finally,  the  lens  is  moved 
to  the  position  where  the  maximum  sig¬ 
nal  was  detected 

In  practice,  because  of  the  low  pass 
nature  of  imaging  systems  and  the  fact 


that  very  low  spatial  frequencies  are 
insensitive  to  focus,  it  is  only  neces¬ 
sary  to  monitor  a  band  of  spatial  fre¬ 
quencies.  The  maximum  sensitivity  to 
change  in  focus  occurs  at  high  spatial 
frequencies,  but  the  signal  alrength  is 
greater  at  lower  frequencies. 

Reference  4  gives  additional  details 
of  automatic  focus  detection  using  the 
DEFT  sensor. 

Motion  detection: 
following  Fourier  phase 

For  motion  deleclion.  the  DEFT  cam¬ 
era  utilizes  the  spatial  shifting  theor¬ 
em  of  Fourier  transforms.  This  theorem 
states  that  a  pure  translation  of  a  pic¬ 
ture  in  real  space  leaves  the  magni 
tude  of  Fourier  component*  unchanged, 
but  alters  their  phase  by  an  amount 
proportional  to  the  magnitude  of  trans¬ 
lation.  Therefore,  by  monitoring  the 
phase  of  a  Fourier  component,  you  can 
delect  any  motion  of  the  picture.  In¬ 
dead.  because  the  DEFT  device  is  a 
two-dimensional  Fourier  transformer, 
you  can  also  determine  the  direction  of 
motion  with  it. 

To  test  this,  we  projected  a  periodic 
grid  pattern  on  the  DEFT  sensor  (see 
Figure  2A)  and  monitored  the  phase  of 
its  fundamental  Fourier  component. 
The  results  are  shown  in  Figure  3. 
which  illustrates  that  the  phase 
changes  linearly  and  undergoes  a 
360-degree  shift  at  2.1  millimeters. 
This  is  exactly  the  periodic  length  of 
the  grid  image  on  the  sensor. 

We  have  also  developed  a  simple  al¬ 
gorithm.  explained  in  reference  5.  that 
yields  the  velocity,  as  a  function  of 
time,  of  an  object  moving  against  a 
complicated  background. 
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The  DEFT  sensor  can  be  used  to  sim¬ 
plify  thp  detection  of  image  features 
such  as  a  key  word  in  a  page  of  text  or 
a  man-made  ob|ect  in  an  aerial  photo¬ 
graph  Detection  of  these  features  with 
an  optical  imager  would  require  a 
search  of  the  field  of  view  for  the  nght 
feature  That  is.  each  segment  of  the 
image  where  the  feature  might  appear 
must  be  correlated  with  a  reference 
template 

Using  the  DEFT  aensor.  it  is  possible 
to  detect  the  desired  feature  in  the 
Fourier  transform  of  the  image  rather 
than  in  the  image  itself,  because  every 
feature  has  a  unique  Fourier  trans¬ 
form  The  advantage  in  using  the 
transform  arises  from  the  Fourier 
transform  apatial  ahifting  theorem 
That  theorem  apecifies  that  no  matter 
where  a  feature  is  within  an  image,  the 
magnitude  of  the  Fourier  transform  of 
the  feature  will  always  be  cantered  at 
the  origin  of  the  Fourier  plane  There¬ 
fore.  it  is  only  necessary  to  correlate 
the  feature  template  once  at  the  origin 
In  situations  where  the  feature  may 
be  rotated  as  well  as  translated,  the 
template  can  be  rotated  around  the  or¬ 
igin  of  the  Fourier  plane  to  determine 
the  best  match.  In  contrast,  aearching 
for  a  rotated  and  translated  feature  in 
the  image  directly  requires  rotation  of 
a  template  around  the  center  of  every 
segment  of  the  image  where  the  fea¬ 
ture  might  appear.  □ 
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APPENDIX  C 

CONTROLLED  ILLUMINATION  CONCEPT 


The  concept  outlined  on  the  following  pages  represents  one  of  the  more 
sophisticated  applications  of  controlled  illumination.  The  technique  used  to 
generate  the  illumination  pattern  is  unique;  space  coding  the  pattern  helps 
minimize  the  amount  of  data  which  must  be  stored  and  processed. 

Positive  aspects  of  this  approach  include: 

•  no  mechanical  scanning, 

t  simultaneous  large  area  illumination,  and 

•  with  a  laser  source,  the  illuminating  array  can  be  well  controlled. 

Negative  aspects  include: 

•  a  high-speed  electronic  shutter  must  be  developed, 

•  as  a  bistatic  system,  obscuration/shadowing  cannot  be  avoided,  and 

•  large  amounts  of  image  data  must  be  stored  and  processed. 

The  latter  point  is  worthy  of  further  discussion.  Consider  the  case 
where  the  illumination  array  is  confined  to  a  symmetrical  M  x  M  pattern,  where 
to  simplify  later  processing,  M  is  a  multiple  of  2.  Consider  the  case  where  a 
128  x  128  array  is  used  (M  =  27).  Each  illumination  spot  can  then  be 
uniquely  identified  with 

N  =  2(1  +  log2M)  =  16  images  =  2*  images 

Proper  use  of  the  illuminating  array  could  reduce  this  by  a  factor  of  2,  but, 
for  generality,  it  will  be  maintained  here.  To  adequately  r  jive  all  M  x  M 
spots  should  require  at  least  a  factor  of  4  improvement  in  resolution  over  the 
number  of  elements  to  be  viewed.  Therefore,  each  Image  would  of  necessity 
have  to  be  composed  of 
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4xM  x  4xM  elements  (512  x  512) 


or  2^  x  2?  x  2^  x  2?  =  2*8  pixels.  With  a  grey  level  resolution  of  8  bits 
(28),  effective  spatial  mapping  would  then  require 

24  x  218  x  28  =  230 

bits  of  information.  This  is  a  large  number  of  data  points  to  store  and 
process.  If  a  reasonable  2  computer  operations/bit  of  information  is  assumed, 
and  the  desired  coordinate  map  will  be  generated  in  1  second,  then  a  computa¬ 
tional  rate  of 


2  x  230  _  z31 

1 

operations/second,  or  in  excess  of  two  billion  operations/second  will  be 
required.  This  is  at  least  three  orders  of  magnitude  faster  than  systems 
currently  available. 

Discussions  with  one  of  the  authors  has  ascertained  that  the  problem  is 
reduced  in  complexity  by  using  binary  image  coding  and  then  creating  pseudo¬ 
images  for  subsequent  processing.  The  operating  rate  would  then  reduce  to 
223  operations/second,  or  a  minimum  of  10-MHz  processing  rate.  Such  rates 
are  above  the  1-MHz  rate  available  with  today's  minicomputer  technology,  and 
to  be  able  to  use  this  concept  demands  the  use  of  considerable  preprocessing 
and  hardwired  computational  techniques.  Both  are  feasible  with  today's 
technology. 

Effective  technological  utilization  requires  high-speed  accurate  coding 
of  the  illumination  array.  To  reach  practical  data  acquisition  rates  demands 
that  the  masking  used  for  coding  be  accomplished  electro-optical  ly.  Mechani¬ 
cal  shuttering  is  not  fast  enough,  but  it  is  not  clear  that  an  effective 
electro-optic  shutter  can  be  developed.  Work  on  developing  such  a  shutter  is 
in  progress. 
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Abstract.  A  method  Is  described  lor  high-resoiution  remote  three 
dimensional  mapping  ol  an  unknown  end  arbitrarily  complex  surface  by 
rapidly  determining  the  three-dimensional  locations  ol  MxN  sample 
points  on  that  surface.  Digital  three-dimensional  (3-D)  locations  delining 
a  surface  are  acquired  by  (1)  optically  transforming  a  single  laser  beam 
into  an  (expanded)  array  ofWxN  individual  iaser  beams,  (2)  illuminating 
the  surface  ol  interest  with  this  array  ol  MxN  (simultaneous)  laser 
beams,  (3)  using  a  programmable  electro-optic  modulator  to  very  rapidly 
switch  on  and  ofl  specified  subsets  of  laser  beams,  thereby  Illuminating 
the  surface  ol  interest  with  a  rapid  sequence  ol  mathematical  patterns 
(space  code),  (4)  image  recording  each  ol  the  mathematical  patterns  as 
they  rellect  ofl  the  surface  using  (a)  a  wavelength-specific  optically 
filtered  video  camera  positioned  at  a  suitable  perspective  angulation 
and  (b)  appropriate  image  memory  devices,  (5)  analyzing  the  stored  im 
ages  to  obtain  the  3  D  locations  of  each  of  the  MxN  illuminated  points 
on  the  surface  which  are  visible  to  the  camera  or  Imaging  device,  and  (6) 
determining  which  ol  the  laser  beams  in  the  array  do  not  provide  reflec¬ 
tions  visible  to  the  imaging  device  Space  coding  of  the  light  beams 
allows  automatic  correlation  of  the  camera  image  (of  the  reflected  spot 
pattern  from  the  surface)  with  the  projected  laser  beam  array,  thus  en¬ 
abling  triangulation  of  each  illuminated  surface  poini  Whereas  ordinary 
laser  rangefinders  aim  and  project  one  laser  beam  at  a  time  and  expect 
to  receive  one  laser  beam  reflection  (bright  dot  Image)  at  a  time,  the  pres 
ent  system  is  optical  (nonmechanical  and  vibration-free)  and  can  collect 
all  the  data  needed  for  high-resolutlon  3-D  topographic  mapping  (of  an 
MxN  sample  of  surface  points)  with  the  projection  ot  as  few  as  t  4 
log?N  light  patterns.  In  some  applications  involving  a  rapidly  changing 
time-dependent  environment,  these  1  4  log?N  patlerns  can  be  projected 
simultaneously  in  different  wavelengths  to  allow  virtually  Instantaneous 
data  collection  tor  a  surface  topography  The  hardware  and  software 
used  in  determining  the  (x,y,z)  location  of  each  surlace  dot  can  be  made 
highly  parallel  and  can  handle  noise  as  well  as  multiple-grazing  reflec¬ 
tions  ol  laser  beams.  In  common  with  many  other  active  rangelinder 
devices,  the  proposed  method  Is  unambiguous  In  determining  the 
topography  of  all  nonspecular.  Illuminated,  and  visible  surlaces  within 
Its  operating  (stereo)  range,  is  simple  to  set  up  and  calibrate,  requires  no 
a  priori  knowledge  of  the  object  to  be  Inspected,  has  a  high  signal-to- 
noise  ratio,  and  is  largely  Insensitive  to  material  textures,  paint 
schemes,  or  epidermal  properties  that  mask  surface  features  to  Inspec¬ 
tion  by  passive  topographic  devices 

keywords-  three-dimensional  (3  D)  Imaging .  automated  replication ,  robot  vision; 
topographic  mapping  pattern  recognition;  artilicial  Intelligence,  photogram 
metry.  electro-optics,  laser;  Imaging. 
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I.  INTRODUCTION 

Interest  in  robot  vision  has  greatly  increased  recently  because  pro¬ 
viding  a  vision  capability  to  an  industrial  robot  would  enhance  its 
versatility  and  generic  utility  in  a  factory/assembly  environment. 
For  example,  computer-aided  manufacturing  may  be  improved  if  a 
real-time  fuily-three -dimensional  (3-D)  computer-aided  inspection 
system  were  on  line 
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A  standard  videocamera  for  robot  vision  provides  a  two- 
dimensional  image  which  usually  contains  insufficient  information 
for  a  detailed  three-dimensional  reconstruction  of  an  object.  (This 
is  not  always  a  problem,  however,  if  the  objects  of  interest  in  the 
robot/inspection  environment  can  be  mathematically  defined 
and/or  Ubeled  in  advance.) 

To  obtain  the  additional  information  needed  for  three- 
dimensional  mapping  of  objects  with  complex  surface  shapes,  a 
scene  can  be  analyzed  passively  by  stereo  photogrammetry  or  ac¬ 
tively  with  rangefinders  and  coded  illumination  Passive 
stereophotogrammelry  generally  requires  a  human  operator  to 
determine  corresponding  scene  positions  in  different  photographs. 
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and  is  therefore  too  slow  for  real-lime  applications  Automated 
passive  stereophotogrammetry  requires  considerable  analysis  ,_4 
Methods  of  actively  interrogating  a  scene  (by  applying  various 
kinds  of  light  to  the  scene)  have  been  used  in  recent  years  Laser 
rangefinders’-6  project  one  beam  onto  the  scene  at  any  instant;  thus 
there  is  no  difficulty  in  correlating  the  illuminated  position  in  the 
scene  with  its  images  in  each  of  two  cameras.  Ah  hough  this  method 
requires  as  many  “images”  as  there  are  sample  points  in  the  scene, 
very  rapid  sequential  laser  rangefinders  may  soon  be  possible  5 
Holography  requires  the  interference  of  phase  coherent  hght  beams 
(one  beam  scattered  off  the  scene  and  one  reference  beam),  but  the 
scene  must  be  free  of  vibrations,  and  to  extract  numerical  data  is 
ofien  difficult.  Three-dimensional  information  has  also  been  ob¬ 
tained  by  illuminating  the  scene  from  different  directions*’11  and 
by  applying  light  grids12,15  and  light  stripes  14  15  The  light  stripe 
method  appears  to  have  been  adapted  recently  for  commercial  use 
to  create  3-D  busts  and  sculpture." 

The  system  described  in  this  paper  analyzes  a  sequence  of  laser 
dot  patterns  which  are  rapidly  projected  onto  a  surface  and  viewed 
from  one  or  more  suitable  perspectives.  An  example  of  a  system 
consists  of 

(1)  a  laser  beam  array  generator: 

(a)  a  single  laser, 

(b)  a  lens  and  shearing  plate  assembly  that  expands  and  parti¬ 
tions  the  primary  laser  beam  into  a  two-dimensional 
(usually  rectangular)  array  of,  say,  MxN  separate  laser 
beams  (where  M  and  N  are  typically  about  128). ,9-20 

(c)  a  spatially  programmable  electro-optic  light-modulating 
device  to  sequence  the  MxN  beams  through  several  (for 
example)  binary-encoded  patterns; 

(2)  an  optical  image  recorder: 

(a)  one  or  more  video  cameras  (with  wavelength-specific  op¬ 
tical  filters),  each  of  which  captures  in  digital  form  (or 
transmits  in  analog  form  to  an  A/D  converter)  the  image 
of  each  coded  pattern  as  reflected  from  the  surface  and 
seen  from  the  particular  camera  perspective, 

(b)  a  device  to  synchronize  all  the  TV  cameras  with  the  pat¬ 
terns  generated  by  the  electro-optic  device. 

(c)  a  buffer  storage  device  to  hold  a  sequence  of  images, 
and/or  a  device  to  rapidly-  transfer  image  data  to  a  com¬ 
puter; 

(3)  software: 

(a)  software  which  rapidly  decodes  the  sequence  of  TV  im¬ 
ages  and  calculates  the  position  (x.y.z)  of  the  surface  at 
each  visible  dot, 

(b)  a  software  warning  capability  which  can  automatically 
detect  inconsistent  or  incomplete  data  (e.g.,  from  incot- 
rectly  pointed  TV  cameras)  and  can  suggest  corrections  to 
the  operator, 

(c)  software  for  image  processing  and  error  detection 
(possibly  with  an  extra  parity  bit  image), 

(d)  software  for  interpolation  between  surface  points  to  ob¬ 
tain  a  continuous  surface, 

(e)  software  for  fully-threc-dimensional  pattern  recognition, 
motion  detection,  etc.,  depending  on  application. 

An  array  of  User  beams,  subsets  of  which  can  be  turned  on  and 
off  by  an  electro-optic  shutter  under  computer  control,  can  be 
perceived  as  an  "active  camera.”  The  present  paper  then  discusses 
a  modified  method  of  stereophotogrammetry  using  one  active 
camera  and  at  least  one  “passive  camera”  rather  than  two  passive 
cameras  as  in  conventional  stereophotogrammetry.  (See  also  the 
preliminary  accounts  of  this  system  2I-22)  More  than  one  passive 
camera  may  be  used  to  view  the  projected  patterns  from  more  than 
one  viewing  angle  if  the  surface  of  interest  is  rough  or  convoluted. 
Systems  using  several  active  cameras  (projectors  of  User  beam  ar¬ 
rays  each  at  a  selected  wavelength)  and  several  passive  cameras  can 
also  be  used  for  those  applications  requiring  both  low-resolution 
global  mapping  of  Urge  surfaces  and  simultaneous  high-resolution 
mapping  of  selected  areas,  or  where  simultaneous  viewing  of  multi¬ 


ple  surfaces  is  desired,  or  if  the  surfaces  of  interest  are  changing  in 
time 

The  active-passive  camera  system  has  several  advantages  over 
strictly  passive  camera  systems.  Industrial  pans  of  varied  composi 
lion,  material  reflectivity,  material  finish,  and  textural  or  paint 
combinations,  can  produce  artifacts  or  ambiguities  when 
straightforward  passive  video  imaging  is  used;  interpretation  dif¬ 
ficulties  and  boundary  definition  for  pattern  recognition  may 
become  difficult  especially  when  convolutions  occui  in  complex 
shapes.  The  projection  of  discrete  beams  creates  unambiguous 
reflective  peaks  (bright  dots)  on  the  object  surface  that  are  largely 
independent  of  surface  characteristics,  and  are  detectable  despite 
peak  intensity  variation  between  dots  on  mixed  textural  surfaces. 
Natural  protective  coloration  of  biological  specimens  in  their 
natural  habitat  could  mask  passive  analysis  but  are  clearly 
measurable  using  active  sensing. 

The  projection  of  a  User  beam  array  onto  a  surface  produces  a 
dot  pattern  image  in  a  viewing  camera.  With  highly  convoluted  sur¬ 
faces,  the  dot  images  may  appear  much  less  ordered  than  the 
original  beam  array.  Space  coding,  however,  tags  each  column  in 
the  laser  beam  array  so  that  the  beam  array  column  of  each  dot 
seen  on  the  surface  is  uniquely  identified  no  matter  how  random¬ 
ized  the  dot  images  may  become.  Thus  the  reflected  dot  pattern 
(passive  image)  can  be  correlated  automatically  with  the  original 
beam  array  projection  pattern  (active  image)  to  permit  point-by¬ 
point  trianguUtion  of  the  sample  points  on  the  3-D  surface. 

Beam  reflections  (bright  dots)  hidden  from  the  passive  camera 
sensor  can  be  “detected”  as  missing  by  the  software.  This  is  done 
by  taking  attendance,  that  is,  by  matching  the  MxN  discrete 
beams  projected  with  those  beams  whose  reflections  are  imaged. 
Knowledge  of  which  beams  are  not  imaged  provides  the  feedback 
information  needed  to  reposition  a  (passive  or  active)  camera  dur¬ 
ing  automatic  topographic  scanning  of  a  3-D  object. 

Various  stripe  projection  methods  can  also  be  space  coded  but 
requite  somewhat  more  analysis  to  provide  information  than  does 
the  discrete  dot  (beam  array)  method  described  here. 

A  space  code  for  an  array  of  beams  arranged  in  M  rows  and  N 
columns  reduces  the  number  of  images,  1,  necessary  for  correlating 
all  light  spots  seen  on  the  surface  to  I  =  I  4  log2N  (compared  with 
I  *  M  x  N  for  a  laser  scanner),  where  N  is  also  the  number  of  col¬ 
umns  of  the  electro-optic  shutter  which  can  be  individually 
switched.  For  convenience,  the  value  of  N  is  usually  chosen  lobe  a 
power  of  iwo. 

2.  MATHEMATICAL  METHOD 

When  an  array  of  laser  beams  illuminates  a  surface,  at  least  some  of 
the  illuminated  surface  positions  can  be  imaged  by  an  image  record¬ 
ing  device  (e.g.,  a  video  camera)  at  a  suitable  perspective.  The 
passive  image  plane  then  contains  a  Urge  number  of  bright  dots 
each  caused  by  a  light  ray  connecting  an  illuminated  position  on  the 
surface  with  the  passive  camera  focus.  The  information  in  the 
passive  image  pUne  (that  is,  the  projection  of  some  of  the  3-D  sur¬ 
face  spots  onto  the  2-D  image  plane)  is  by  itself  insufficient  to 
determine  the  3-D  positions  of  the  illuminated  spots  on  the  surface. 

Suppose  «n  array  of  User  beams  diverges  from  some  focal  point 
or  laser  source  and  passes  through  a  transparent  active  image  (shut¬ 
ter)  plane.  If  (1)  a  particular  User  beam  (identified  by  its  intersec¬ 
tion  with  the  iransparem  active  image  pUne)  illuminates  a  spot  on 
the  surface  of  interest,  and  (2)  the  image  of  that  spot  can  be  located 
in  the  passive  camera  plane,  then  the  3-D  spatial  position  of  the  sur¬ 
face  spot  can  be  determined  (just  as  in  stereophotogrammetry)  pro¬ 
vided  the  homogeneous  4x4  transformation  matrices  (containing 
parameters  of  orientation,  displacement,  perspectivity,  scaling, 
etc.)  are  known  for  the  active  camera  0*ser  source  and  shutter)  and 
the  passive  camera. 

The  passive  camera  image  (x*,y*)  of  a  point  (x.y.z)  in  the  scene  is 
given  by  the  perspective  transformation22’24 

<T„-TI4x»)x  4  (TjrTJ4x*)y  4  (Tji-T^t*)! 
fTarT^*)  -  0 


-*  a4rT4*><)  ■=  0  (2) 

If  ihc  scene-to-image  transformation  matrix  T  is  inown,  we  have 
for  each  User  dot  visible  to  the  TV  camera  the  known  quantities 
Tijt  x*.  y*  and  two  equations  for  the  three  unknowns  x,  y,  2.  We 
need  one  more  equation.  Suppose  our  laser  beam  uny  passes 
through  an  electro-optic  shutter,  to  that  the  intersection  of  a  beam 
with  the  shutter  plane  has  a  unique  position  (u.w)  in  that  plane. 
Then  the  laser  beam  array  can  also  be  described  in  terms  of  a 
perspective  transformation 

<Llt-LMu)x  4  (L2,-L24u)y  4  (LjrL^ujz 

4  (L^-L^u)  =  0  (3) 

<L|j-L,4w)x  4  (L22-L24w)y  4  (Ljj-LjjW)? 

4  (L^-L^w)  =  0,  (4) 

where  L  is  the  scene-to-laser  transformation  matrix,  (u.w)  identifies 
the  particular  beam  in  the  shutter  plane,  and  (x.y.x)  is  (as  before) 
the  (unknown)  position  on  the  surface  (in  the  scene)  that  the  laser 
beam  hits. 

We  apply  space  coding  to  associate  with  each  image  point  (x*,y*) 
the  value  u  of  the  corresponding  laser  beam.  We  then  have  the 
given  quantities  Tjj,  Lj.,  x*.  y*.  u  and  solve  Eqs.  (I),  (2),  (3)  above 
for  the  three  unknowns  x.  y,  2  provided  the  equations  are  non¬ 
singular.  Thus  for  each  image  point  (x*,y*)  we  can  obtain  the  cor¬ 
responding  surface  position  (x,y,2). 

The  camera  equations  by  themselves  give  two  equations  for  three 
unknowns  and  thus  determine  only  the  ray  from  the  scene  to  the 
camera  The  first  equation  for  the  User  perspective  transformation 
(with  u  given  by  the  space  code)  provides  the  plane  which  intersects 
the  ray  to  the  camera.  Clearly,  a  well-conditioned  solution  for  x.  y, 

2  requires  that  the  laser  and  camera  parameters  (in  particular,  the 
User  and  camera  positions  and  the  space  coding  pUnes  u  «=  const) 
are  such  that  the  solution  rays  from  the  camera  are  not  nearly 
parallel  to  the  mathematical  planes  determined  by  Ljj  and  u  =  con¬ 
stant.  Well-conditioned  solutions  (accurately  determined  positions) 
should  be  obtainable  as  long  as  the  points  in  the  scene  are  noi  ex¬ 
tremely  distant  from  the  camera  User  system  (where  all  distances 
are  measured  relative  to  the  camera-laser  separation  distance). 
Once  we  find  x,  y,  2,  we  can  calculate  w  from  the  last  equation  so 
that  we  can  Uter  determine  which  User  beams  (u.w  )  have  been  imaged. 

3.  SPACE  CODING 

We  now  describe  the  space  coding  technique.  Suppose  *t  have  an 
MxN  array  of  laser  beams  (M  rows,  N  columns)  which  pass 
through  an  electro-optic  shutter  plane  u.w,  Let  the  centroid  of 
beam  (n,m),  where  1  s  m  <  M  and  1  s  n  <  N,  intersect  the  shut¬ 
ter  plane  at  some  position  (up,wnl1,  such  that  n-1  s  un/a  <  nand 
m  l  s  Wjj/b  <  m,  where  a  is  thedisunce  between  the  midlines  of 
the  adjacent  columns  of  the  laser  beam  array  and  b  is  the  distance 
between  the  midlints  of  the  adjacent  rows  of  the  User  beam  array. 
With  this  definition,  that  beam  of  the  laser  beam  array  which 
passes  through  the  shutter  plane  at  position  (u.w)  is  identified  with 
the  unique  integer  pair 

(n,m)  =  (I  4  flr(u/a),  I  4  flr(w/b)) ,  (5) 

where  flr(x)  «  largest  integer  contained  in  real  number  x. 

We  design  the  electro-optic  shutter  to  have  N  separately  con¬ 
trollable  columns  (one  for  each  column  of  the  User  beam  array)  so 
that  if  we  apply  an  electric  signal  to  one  of  N  input  wires,  say  wire 
n,  the  domain  |(u,w)  (n-l)a  <  u  <  na|  of  the  User  shutter  will 
become  opaque  With  such  a  shutter  we  can  control  which  beams 
of  the  laser  beam  array  are  transmitted  and  which  are  blocked,  and 
in  this  way  encode  patterns  in  the  array  of  transmitted  beams.  By 
projecting  a  sequence  of  I  4  logjN  binary  User  beam  patterns,  we 
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can  determine  uniquely,  for  any  dot  image  seen  in  the  passive  image 
pUnc,  the  address  n  of  the  shutter  column  of  the  corresponding 
User  beam. 

As  an  example,  suppose  we  have  a  200x16  User  beam  array 
passing  through  an  electro-optic  shutter  with  16  controIUble  col 
umns  Ubeled  by  I  4  flr(u/a).  We  then  seqt  -ntially  project  (and 
image)  the  following  patterns:  (1)  the  enure  User  beam  array,  (2) 
the  higher-numbered  half  of  the  array  (columns  16  through  9 
transparent;  columns  8  through  1  opaque),  (3)  alternate  quarters  of 
the  array  (columns  16  through  13  and  8  through  $  transparent;  all 
others  opaque),  (4)  alternate  eighths  of  the  array  (columns  16,  15, 
12,  II,  8,  7,  4,  3  transparent;  all  others  opaque),  (5)  alternate  col 
umns  of  the  array  (even  columns  transparent;  odd  columns 
opaque).  Altogethet  1  4  log 2N  patterns  must  be  projected  and  im¬ 
aged  for  a  binary  space  code.  In  the  case  just  illustrated,  N  =  16,  so 
that  1  4  log ,(16)  =  5.  Then  for  any  spot  imaged  by  the  camera 
during  the  first  pattern  (all  the  beams),  the  value  of  flr(u/a)  of  the 
corresponding  beam  in  the  laser  beam  array  is  given  by  the  binary 
number  obtained  by  the  appearance  or  nonappearance  of  the  spot 
in  the  images  of  subsequent  patterns.  For  example,  suppose  a  sur¬ 
face  spot  appears  in  the  passive  camera  images  of  patterns  1,2,  and 
4  but  not  in  patterns  3  and  5.  Then 

flr(u/a)  =  10102  =  10 

(corresponding  to  its  appearance  or  nonappearance  in  patterns  2, 3, 
4,  5)  so  that 

n  =  1  4  f)r(u/a)  =  1) 

shutter  domain  =  |(u,w):  I0»  sus  l)i|. 

The  apparatus  can  be  set  up  and  calibrated  so  that  a  more  precise 
position  value  u„  on  the  shutter  plane  is  known  for  each  column  n 
of  the  laser  beam  array.  Thus  once  the  shutter  column  n  «=  1  4 
f)r(u/a)  is  determined,  the  value  of  un  is  known  for  the  laser  beam. 

If  the  transformation  parameters  Tjj  and  L;j  are  known,  we  can 
obtain  (x,y,z)  for  each  (x'.y’.u).  Once  (x,y,2)  is  known,  w  can  be 
found  with  the  last  equation  Any  beam  (u,w)  not  visible  to  the 
camera  can  be  found  by  listing  all  the  projected  beams  and  then 
eliminating  from  the  list  those  that  art  visible. 

In  Fig.  1  we  illustrate  the  space  coding  for  a  6x8  array.  Only- 
four  spots  on  the  scene  are  visible  to  the  camera  in  this  example. 
The  laser  shutter  column  corresponding  to  each  image  point  is 
found  in  Table  1  from  the  data  of  Fig  1 . 


TABLE  I. 


Image  point 

Binary  number 

flr(u/a) 

n 

A 

0 00 

0 

1 

B 

Oil 

3 

4 

C 

110 

6 

7 

D 

010 

2 

3 

The  space  code  method,  which  allows  us  to  use  one  passive 
camera  and  one  active  camera,  should  be  compared  with  the  case  of 
stereophotogtammelry  in  which  one  must  use  two  cameras  and 
know  the  corresponding  transformation  matrices  Tj.  Tjj  and  the 
corresponding  image  points  (xa',y,*),  (*b*.yj,‘)  for  fhr  same 
unknown  point  (x,y,2)  of  the  scene,  hence  four  equations  arc 
solved  for  the  three  unknowns  x,  y,  1  with  inconsistencies  treated 
by  least  squares. IJ-J4 

4.  OBTAINING  THE  TRANSFORMATION 
PARAMETERS 

In  a  laboratory  the  camera  and  laser  beam  projector  can  be  fixed 
and  the  parameters  Tjj  and  Ljj  measured  directly. 

In  portable  operation,  as  for  example  a  medical  or  industrial  ap- 
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Fig.  1.  Th«  Array  of  la  tar  beams  Is  apace  coded  by  the  sequence  ol 
patterns  shown  at  the  left,  which  Is  obtained  by  controlling  the 
opaquaness  ot  the  columns  ot  the  ahutlar  plana.  The  Intersections 
of  tha  la  sar  beams  with  the  scene  provide  bright  spots  on  the  scene 
which  can  be  Imaged  from  a  aultabla  perspective  by  (say)  a  TV 
camera  (provided  thera  are  no  obstructions  In  the  tine  ot  sight  be 
tween  the  scene  and  the  camera).  The  TV  camera  Is  synchronized 
with  the  ahuttar  ao  that  lor  each  protected  pattern  a  TV  Imaga  Is  ob 
talnad.  In  the  Illustration,  the  TV  Image  sees  lour  spots  ol  the  scene 
corresponding  to  lour  lasar  beams.  The  columns  ot  the  ahuttar 
plana  through  which  thesa  beams  passed  can  be  determined  by  the 
spaca  coding.  Point  B  Is  aaan  In  pattam  a  and  pattern  4  but  not  In 
pattarn  2  and  ao  conalatas  with  ahuttar  column  1  4  OII2  1  4  3  c  4. 


plication,  it  may  be  necessary  to  set  up  and  disassemble  the  equip¬ 
ment  periodically  or  per  object  to  be  mapped,  so  a  rapid  calibration 
scheme  is  needed  which  a  nonexpen  can  immediately  use. 

One  simple  calibration  method  is  to  insen  into  the  scene  a  prism 
whose  faces  are  known  planes.  To  find  the  T  transformation 
matrix,  we  locate  six  vertices  of  the  prism  in  the  passive  image 
plane,  so  that  we  know  the  values  (xk*,yk#)  and  (xk,yk  ,zk)  for  i  * 

1 ....  ,6  We  can  then  set  T*,  -  I  and  solve  eleven  equations  for 
the  eleven  unknowns  T,j  of  Eqs.  (I)  and  (2).JJJ<  To  obtain  the  L 
transformation  matrix,  we  project  the  coded  laser  beam  array  and 
image  the  prism.  If,  say,  N  £  7  suitable  points  are  imaged,  we  can 
obtain  coefficients  Lj p  Ljj,  Lj|,  L.4j,  L j ^ of  Et).  (3)  by 
setting  *  I  and  minimizing 

N 

F  “  £  [(Ltt-Ll4uk)xl  4  (LjrLj4uk)yk 

k  =  1 

4  (LJt-Lwuk)zk  4  (L4,-uk)]  J .  (6) 

where  the  subscript  k  refers  to  one  of  the  N  points  imaged  by  the 
passive  camera.  The  values  (xk,yk,zk<uk)  can  be  found  for  each  im 
aged  point  k  because  we  know  the  Tj:  (from  before),  the  image 
position  (xk*,yk*),  the  space  code,  and  (he  planes  of  the  prism.  The 
least-squares  solution  is  obtained  by  solving  the  set  of  seven  linear 
equations 

aF/aijj-o.  (7) 


one  for  each  of  the  seven  unknow-n  Lj  coefficients  A  solution  ex 
ists  if  the  7  x?  matrix  is  nonsingulat.  In  most  cases,  a  nonsingular 
matrix  etccurs  if  the  prism  is  within  the  active-passive  stereo  range 
and  the  N  points  appear  on  more  than  one  face  of  the  prism  In  this 
method,  the  accuracies  of  the  Ly  values  depend  on  the  accuracies 
of  the  Ty  values.  This  automatic  calibration  procedure  does  not 
provide  tne  coefficients  of  Eq.  (4),  so  that  solving  for  the  shut¬ 
ter  row  w  after  x.y.z  has  been  obtained  (that  is,  attendance-taking 
information)  would  not  be  possible.  To  obtain  the  Ljj  of  Eq  (4),  a 
second  automatic  calibration  procedure  involving  a  row-coded 
electro-optic  shutter  can  be  easily  appended 

The  easiest  situation  of  all  would  be  to  do  all  calculations  relative 
to  a  coordinate  system  fixed  to  the  laser  projector  or  the  main  body 
of  the  robot  and  to  sense  independently  the  camera  orientation  and 
other  parameters  with  respect  to  this  reference  frame  (say  by  means 
of  separate  optical,  mechanical,  or  inertial  guidance  systems)  With 
such  tensing  capability,  the  projector  and  the  sensor  can  be  in¬ 
dependently  manipulated  and  need  not  be  limited  to  predeter¬ 
mined,  precalibrated  viewing  positions. 

No  matter  what  calibration  is  used,  the  cameras,  active  or 
passive,  need  not  be  at  the  same  relative  distance  from  the  object  of 
interest  but  can  be  placed  at  any  convenient  locations  within  the 
stereo  range.  The  only  restriction  in  positioning  the  active  and 
passive  cameras  is  that  the  solution  of  Eqs.  (1),  (7).  (3)  must  exist 
and  be  well  conditioned 

5.  OPTICS  FOR  LASER  BEAM  ARRAY  GENERATION 
Laser  light  can  be  expanded  with  a  lens  and  convolved  into  rec¬ 
tangular  grid  patterns  by  a  parallel  arrangement  of  fringe- 
generating  shearing  plates.  However,  for  spatial  encoding  and  con¬ 
venient  light  image  capture  by  videocamera,  an  array  of  lighted, 
discrete  (Gaussian  isolablc)  points  is  desirable  We  now  describe 
a  series  arrangement  of  shearing  plates  to  generate  the  discrete 
(bright)  beam  array  pattern  required. 

The  principle  used  in  creating  a  fringe  array  is  outlined  in  Fig 
2. 1,  J0  Interference  fringes  are  generated  by  front  and  back  reflec¬ 
tion  from  a  shearing  plate.  The  condition  for  the  creation  of  N  in¬ 
terference  fringe  lines  is19 

N  ■=  (2d/X)|tan‘,(Wc/2f1))|sin  2o)/(nJ-sin5o),/J ,  (8) 

where  d  is  the  shearing  plate  thickness,  n  is  the  index  of  refraction 
of  the  plate,  WD  is  the  diameter  of  the  incident  laser  beam,  f,  is  the 
focal  length  of  the  focusing  lens,  X  is  the  wavelength  of  the  laser 
light,  and  o  is  the  incident  angle  of  the  principal  ray. 
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If  fringe  bnes  produced  by  one  plait  art  incident  on  a  second 
partially  silvered  plait  in  such  a  way  lhai  the  plane  of  incidence  of 
the  second  plate  is  perpendicular  to  the  first,  the  fringe  lines  will 
again  be  broken  by  interference  minima,  and  the  light  reflected 
from  the  second  plate  will  constitute  an  expanding  rectangular  ar¬ 
ray  of  lighted  beams  emanating  from  a  common  point.  Thu  light 
array  can  be  recolli mated  and  spatially  filtered  to  produce  a  sharp 
discrete  array  of  laser  light  beams  The  final  array  may  then  be 
caused  to  converge  for  high  resolution  pattern  recognition  onto 
microscopic  objects,  or  to  expand  at  a  convenien  angular 
divergence  to  map  larger  objects.  A  suitable  combination  of  lenses 
is  used.  The  relative  size  of  the  beams  with  respect  to  the  distances 
between  beam  centers  (measured  in  a  plane  perpendicular  to  the  ar 
ray  path,  i.e.,  parallel  to  the  shutter  plane)  remains  constant  as  the 
total  array  size  and  individual  beam  diameters  increase  over 
distance  in  a  diverging  pattern 

A  decided  advantage  of  this  method  to  generate  beams  in  an 
ordered  array  is  that  the  discrete  beams  that  form  the  pattern  are 
the  product  of  diffraction,  the  beams  behaving  as  individual  lasers. 
Patterns  will  not  deteriorate  at  long  range  (in  a  vacuum),  and  can 
(theoretically)  be  converged  to  wavelength  separation  between 
beams  This  is  in  sharp  contrast  to  forming  a  beam  pattern  by  con¬ 
ventional  masking  apertures.  Masks  result  in  diffraction  effects 
that  seriously  limit  the  useful  range  of  the  beam  array  and  its  degree 
of  useful  convergence 

A  laboratory  demonstration  prototype  of  the  basic  hardware 
concept  is  illustrated  in  Fig  3.  The  system  generates  and  projects  a 
programmed  sequence  of  coded  arrays  of  discrete  bright  laser 
beams  onto  an  object  01  specimen  A  2"-diameter  plaster  stone 
model  of  a  human  premolar  tooth  (object)  is  seen  at  the  lower  right 
mounted  on  a  lens  holder.  The  bright  beams  are  generated  optically 


Ftp  3.  Laboratory  demonstration  prototype  ot  Topographic  Map 
ping  System  using  manually  placod  cards  to  Hlustrala  pattern 
changing  lor  space  coding 


from  a  single  linearly  polarized  lasei  source  and  an  optical  arrange 
ment  which  includes  beam  expansion,  two  optical  shearing  plates, 
and  beam  steering  devices  ("pattern  generating  optics"  of  Fig.  3) 
Any  suiublr  laser  wavelength  or  set  of  wavelengths  can  be  used 
Fot  this  demonstration,  the  514.5  nm  line  of  a  model  165  Spectra- 
Physics  argon-ion  cw  laser  was  chosen  The  optical  pattern  may  be 
intermittent  or  continuously  generated  depending  on  the  laser 
source,  configuration,  and  application  requirement.  Multiple 
source  projection  is  possible  provided  that  either  synchronization 
or  selective  wavelength  filtering  is  used  with  the  scene  capture  imag 
ing  device;  for  simulia  irous  multiple  source  projection,  the  imag 
ing  device(s)  must  have  muliispectral  discrimination  capability. 


6.  HARDW  ARE  FOR  BEAM  AR«vAY  CODING 
The  beam  array  once  generated  is  recollimated  as  required  by  col¬ 
limating  optics  (Fig.  3)  and  then  passed  through  a  programmable 
set  of  rectangular  (columnar)  shutters.  The  shutter  assembly  can  be 
constructed  from  mechanically  actuated  slits,  routing  assemblies 
of  rulings,  liquid  crysul  gates,  or  electro-optic  oi  acousto-optic 
crystal  gales.  The  function  of  the  shutter  is  to  eliminate  selected 
columns  (or  rows)  of  beams  from  the  laser  beam  array.  For  an 
electro-optic  shutter,  each  individual  shutler  column  (or  row) 
becomes  transmissive  or  opaque  to  the  projected  light  beams 
depending  on  the  particular  electrical  signal  it  receives.  The  collec¬ 
tive  effect  of  blocking  individual  columns  (or  rows)  dicutes  the 
particular  pattern  being  transmitted.  The  shutter  columns  (or  rows) 
are  controUed  by  b  computer  or  microprocessor  which  drives  the 
shutler  to  transmit  a  programmed  sequence  of  two-dimensional  ar 
ray  patterns  onto  a  surface  of  interest.  The  reflected  image  of  each 
new  beam  array  pattern  from  the  surface  is  recorded  by  one  or 
more  image  acquisition  devices  (such  as  a  videocamera)  The  shut¬ 
ter  and  the  imaging  devices  are  synchronized  by  a  microprocessor . 

The  manually  positioned  (programmable)  coding  shutlers  shown 
in  Fig.  3  are  presented  simply  to  illustrate  the  basic  concept  of  shut¬ 
ter  pattern  reconfigurations  required  for  space  coding  A  larger 
(128x12*)  beam  array  was  masked  by  a  white  card  set  in  a 
holographic  plate  holder  to  create  an  (illustrative)  eight -column 
wide  pattern  The  space  code  sequence  begins  when  an  unshuttered 
(uncoded)  beam  array  is  first  projected  on  the  surface  of  interest 
and  is  recorded  by  a  suitable  videocamera  and  storage  electronics. 
In  this  demonstration  a  standard  closed  circuit  (RS-1 70  compatible) 
General  Electric  model  TE-26  videocamera  was  used  (upper  left 
comer  of  Fig  3)  and  focused  on  the  tooth  model  (seen  in  right 
foreground  of  Fig.  3). 

After  this  ftrsl  view  is  recorded  the  shutter  is  changed  or  recon¬ 
figured.  For  illustrative  purposes,  the  second  shutler  configuration 
may  be  represented  by  the  top  white  card  lying  on  the  optical  table 
in  Fig.  3.  This  shutter,  when  placed  in  the  shutler  holder,  eliminates 
four  consecutive  columns  of  beams  while  allowing  four  consecutive 
columns  of  beams  to  pass  unimpeded.  The  scene  is  again  recorded 
to  obtain  the  second  view .  The  third  shutter,  (bottom  white  card  on 
optical  table,  Fig  7)  replaces  the  previous  shutter  to  produce  a  new 
pattern  code.  Figure  4  is  a  photograph  of  the  video  image  recording 
of  the  third  shutler  pattern  as  projected  on  a  tooth  model  (Com 
pare  this  camera  view  with  pattern  3  of  the  shutter  plane  in  Fig  1 .3 
The  fourth  shutter  (seen  in  the  beam  array  path,  Fig  3)  is  then 
placed  and  the  scene  recorded  to  produce  the  fourth  view 


Fig  4.  An  axample  ol  on*  shutter  coded  pattern  as  projected  on  a 
tooth  model  using  an  •  columnar  (programmable)  abutter. 


7 


Manually  placed  (turners  as  iUusiraied  (Fig  3)  m  the  laboratory 
demonstration  (even  if  engineered  into  a  filmstrip  mechanical 
device)  would  be  cumbersome  and  prone  to  failure.  The  optimum 
method  for  blocking  selected  columns  (or  rows)  is  to  use  a  fued, 
vibratiorOess,  programmable  spatial  light  modulator  (PSLM)  with 
exclusively  electro-optic  components  and  properties. 

A  laboratory  demonstration  model  using  a  128  (vertical)  column 
electro-optic  PSLM  is  seen  in  Fig.  5.  The  PSLM  is  visible  in  the 
center  of  the  figure,  just  below-  the  videocamera  lens  and  above  the 
top  of  the  laser  (Fig.  5).  The  tooth  model  and  support  is  to  the  ex¬ 
treme  right  in  the  figure.  In  this  photo,  the  PSLM  has  been  pro¬ 
grammed  with  four  sets  of  “ON"  (transmisive)  blocks  of  columns 
alternating  with  four  sets  of  “OFF"  (opaque)  blocks.  Each  set  has 
sixteen  successive  columns  “ON"  or  “OFF."  A  polarizing  plaie 
mounted  in  a  metal  frame  appears  just  in  front  of  the  shutter.  Not 
visible  (hidden  by  the  videocamera)  is  an  interface  and 
microprocessor  which  controls  the  PSLM. 

Since  space  coding  image  conservation  made  a  log-base-2  width 
system  desirable  and  since  standard  U.S.  videocameras  of  480 


Fig  6  Laboriiorv  demonstration  breadboard  in  which  an  electro 
optic  programmable  apatial  light  modulator  (PSLM)  is  substituted 
lor  manual  cards  (Mluttra  tad  in  Fig  3|  to  provide  apace  coding .  The 
PSLM  blocked  selected  (pr mprogrammtd)  column*  (or  rows  de¬ 
pending  on  orientation)  in  the  projecting  beam  array  path.  The 
PSLM  (oenter  of  photo  below  videocamera  lens  and  above  laser)  is 
shown  with  16  consecutive  rows  on.  16  off.  ate.,  or  126  columns 
in  which  eight  eats  of  1 6  columns  are  alternately  on  (transmissive) 
or  off  (opaque  or  black).  A  microprocessor  (hidden  by  video- 
earner#  lens)  controls  the  PSLM  pattam.  Any  combination  of  the 
1 26  rndnhduel  columns  in  the  PSLM  may  be  selected  for  transmis¬ 
sion  or  opaqueness  Tooth  model  is  at  right  center  fixed  to  a  mag 
oatic  mount. 


useful  scanned  lines  were  available  in  the  laboratory,  a  total  column 
number  of  128  was  chosen  for  this  demonstration.  The  PSLM 
module  was  constructed  by  Applied  Optics  Corporation  (a  division 
of  ITEK  Corporation)  and  mounted  on  a  glass  plate  with  a  total 
optical  (light  modulating)  area  of  2  x2  inches  square.  The  module 
contained  integral  (embedded)  peripheral  electronics  to  assist  in 
loading  the  -pattern  Each  column  of  electro-optic  material  in  the 
shutter  is  individually  programmable.  When  an  appropriate  voltage 
signal  is  applied  to  this  material  it  will  exhibit  half-wave  plate 
behavior.  Since  the  beams  in  the  array  have  definite  polarization 
derived  from  the  laser  source,  any  beam  passing  through  an 
energized  column  would  be  eliminated  by  an  appropriately  oriented 
analyzing  polarizer  placed  beyond  the  shutter. 

An  interface  was  devised  by  Capt.  Michael  TworeL,  USAF,  so 
that  any  suitable  microprocessor  could  be  used  with  the  PSLM. 
Moreoever,  the  requirement  to  load  the  PSLM  with  128  bits  of 


clocked  serial  data  from  the  microprocessor  is  avoided.  Instead,  a 
16-bit  parallel-load-to-serial-data-out  capability  was  created  where 
eight  16-bit  words  are  loaded  into  the  interface  to  properly  code  the 
128  columns  of  a  selected  configuration.  The  interface  marched  an 
available  Intel  SBC  80/20  single-board  computer  to  the  PSLM 
liquid  crystal  materials  rise  time  of  500  ms  Were  a  faster  PSLM  to 
be  used,  the  existing  interface  could  be  redesigned  with  a  faster 
clock  to  load  a  pattern  in  I  ms.  Two  PSLM  modules  and  interfaces 
were  assembled  in  the  optical  path  so  that  either  vertical  columns  or 
horizontal  row-s,  or  a  combination,  could  be  space  coded.  Since  the 
present  PSLM  module  is  too  slow  for  switching  between  video 
raster  scans,  the  preprogrammed  patterns  were  loaded  into  the 
microprocessor  and  a  manual  binary  switching  method  was  used  to 
reconfigute  the  shutiet.  This  was  sufficient  to  demonstrate  space 
coding  by  a  PSLM  in  an  active  camera.  A  second  generation  PSLM 
can  be  constructed  to  permit  reconfiguring  the  PSLM  columns 
within  (and  synchtonized  to)  the  vertical  blanking  time  of  a  stan¬ 
dard  video  raster  scan,  or  during  the  erase /recovery  cycle  of  s  solid- 
state  imaging  camera.  A  100  gs  reconfiguration  rate  could  be 
achieved  with  little  difficulty  (well  within  the  1.4  ms  of  the  vertical 
blanking  time)  using  PLZT  material,  and  even  I  gs  reconfiguration 
rales  ate  theoretically  possible  for  some  candidate  methods  3V3* 
The  limiting  factors  would  then  be  the  videocamera  image  acquisi¬ 
tion  and  throughpui/erase/recovery  cycle,  and  the  ability  of  a 
suitable  device  to  store  or  process  incoming  data 

7.  BEAM  ARRAY  PROJECTION  ONTO  THE  SCENE 
The  choice  of  optics  required  to  adjust  the  total  area  illuminated  by 
the  beam  array,  as  patterned  by  the  coding  subsystem,  is  deter¬ 
mined  by  the  relative  size  and  distance  of  the  object  from  the  beam 
array  projector,  and  the  resolution  required  for  mapping  the  scene 
or  surface  of  interest.  Either  a  nondiverging,  converging,  or  diverg¬ 
ing  beam  array  may  be  used  to  suit  a  particular  application.  A  con¬ 
verging  optical  arrangement  may  be  used  to  project  a  higher  spatial 
frequency  array  onto  a  very  small  object.  For  example,  a  10 
dot/mm  spatial  frequency  array  (100  «im  dot  separation)  has  been 
projected  onto  a  surface  and  viewed  through  a  microscope  objec¬ 
tive.  A  zoom  lens  may  quickly  and  conveniently  alter  the  spatial 
frequency  and  size  of  the  beam  array  projected  field  for  multipur¬ 
pose  uses,  or  for  changing  from  wide  area  to  detailed  scanning. 
Suitable  rotatable  and  gimbal-mounfed  optical  devices  may  be  used 
to  accurately  conirol  the  direction  of  beam  array  projection  The 
laser  beams  in  the  array  are  focused  at  infinity  by  the  beam  array 
generating  optics.  The  laser  beam  array  will  project  through  space 
for  a  considerable  distance  until  impeded  by  an  object  or  surface. 

8.  IMAGE  ACQUISITION 

A  videocamera  records  from  a  suitable  perspective  the  images  of 
the  sequence  of  coded  beam  array  patterns  reflected  off  the  surface 
of  interest.  In  Fig.  6,  a  full  (128  x  128)  pattern  is  intercepted  by  a 
2"-diameter  plaster  premolar  model  which  was  bordered  by  black 
rubber.  A  General  Electric  TE26  vidicon  camera  with  a  GE  f/2.5  to 
22  and  20  to  80  mm  tele-zoom  was  used  to  record  the  pattern.  II 
luminatkm  of  the  object  was  approximately  10  gW/cm3  by  514.5 
nm  laser  light .  Ambient  daylight  fluorescent  lighting  was  present.  A 
*  I0X  selective  filter  was  used  in  from  of  the  telephoto  lens.  The 
image  was  recorded  on  an  I  VC  I"  reel-to-reel  video  recorder  and 
played  back  on  a  black  and  white  19"  monitor.  A  35  mm  still 
camera  recorded  this  photograph  (Fig.  6)  during  playback  A  dot  in 
the  lower  left  comer  (beyond  the  model)  is  an  artifact  in  the 
monitor.  Approximately  three  thousand  sample  points  are  visible 
Figure  7  is  photographed  in  real  time  in  the  laboratory.  Note  am 
bient  lighting  levels  seen  in  Figs.  5  and  7.  A  two  "ON,"  two 
“OFF"  alternating  pattern  was  being  projected 
Figure  8  is  another  image  of  the  object  shown  in  Figs.  6  and  7  In 
this  image  the  pattern  was  configured  with  alternating  eight  col 
umns  on  and  eight  columns  off.  Variations  in  dot  brightness  (inten 
sity)  and  adjacent  dot  to  dot  spacing  are  caused  by  surface  cur 
vatures  of  the  object. 
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Fig  6  Two-lnch-diameter  bicuspid  tooth  plaster  modal  llluminatad 
by  full  beam  array  pattarn  (126x128)  Approximately  3000  dot 
reflections  art  visible  Photo  was  taken  of  video  monitor  while 
videorecording  playback  Object  was  Illuminated  with  a  514  5  nm 
argon  laser  In  a  daylight  fluorescem  lighted  laboratory 
Videocamera  had  a  aelectlve  wevelength  inter  In  Iron!  ol  Its  lens  to 
remove  ambient  lighting  (black  cylinder  housing  on  videocemera 
lens  is  seen  In  top  letl  ol  Fig  5) 


Fig  7.  Photograph  ol  videomonitor  viewing  In  reel  time  e  two-on 
two  oil  consecutive  pattern  projected  onto  the  tooth  model  Am 
blent  lighting  in  laboratory  Is  used  lor  this  exposure 


Figures  9  and  10  show  digitized  images  of  a  prism  played  back 
from  compuiei  tape  onto  a  Comlal  display  unit.  Figure  10  shows 
the  passive  image  of  a  laser  beam  anay  pattern  coded  by  a  PSLM 
and  reflected  off  the  prism 

For  a  standard  composite  video  signal,  a  raster  scan  of  each  im 
age  is  completed  in  I /30th  second  During  Ihe  interval  between  the 
end  of  the  preceding  raster  scan  and  the  beginning  of  a  new  scan, 
the  electro-optic  shutter  reconfigures  A  microprocessor,  minicom 
pulei,  or  clock  synchronizes  the  videocamera,  beam  array  shutter, 
and  video  recording  device  A  video  recorder  or  video  disk  system 
may  be  used  to  initially  record  the  images  An  alternative  method 
would  be  solid-state  storage  of  individual  video  images  on  charge 
coupled  device  (CCD)  or  tandom  access  memory  (RAM)  chips  with 
high  speed  sequential  or  parallel  analog  to  digital  (A/D)  conver¬ 
sion.  A  512  xSI2  x8  pixel  array  from  a  I /30th  second  raster  scan 


Fig  6  Photograph  ol  seme  objact  as  in  Fig.  6  but  with  the  baam  ar¬ 
ray  now  coded  by  Ihe  PSLM  Into  one  ol  the  binary  patterns  A  total 
ot  eight  different  binary  patterns  are  sequentially  protected  lo  com 
p lately  code  a  128  x  126  beam  array  and  tag  each  array  column  tor 
Identification  In  the  Image 


Fig  •  White  light  digitized  Image  ol  prism  block  displayed  on  a 
computer  graphics  terminal  alter  playback  trom  digital  (computer) 

t»P« 


mas  be  stored  in  digital  form  (using  commercially  available  deuces) 
with  each  point  mutually  addressable 

For  a  16,000  point  topographic  spatial  map.  a  total  of  eighi 
coded  images  is  required.  Total  storage  time  is  thus  8/J0ih  second 
(  *  267  ms),  and  eight  images  must  be  stored  for  processing  W  hi)e 
adequate  for  many  applications,  faster  speeds  are  often  desirable. 
If  charge-injection  device  (CID)  or  CCD  direct  imaging 
technologies  are  used,  then  the  speed  of  total  scene  capture  (that  is, 
eighi  views  for  a  16.000  point  3-D)  might  be  extremely  rapid  A  two 
millisecond  time  cycle  is  reasonable  for  acquisition  of  all  data  need 


Fk)  10  Digitized  Image  ol  the  urn  prism  Mock  (displayed  on  a 
computer  graphics  terminal)  with  the  128  x  128  laser  beam  array 
shown  coded  by  a  pattern  ot  the  128-column  PSLM 


ed  to  define  the  three-dimensional  coordinates  (x.y.z)  of  a  beam  ar¬ 
ray  projection  such  as  the  one  on  the  bicuspid  tooth  model  shown 
in  Fig  6  A  512  x  512  C1D  or  CCD  anay  main*  would  have  to  be 
developed  for  rapid  acquisition  of  data,  in  which  the  output  would 
email  simultaneous  (parallel)  output  of  rows  or  blocks  of  rows  of 
the  array  (rather  than  using  the  RS-170  output  rate  serial-scan 
methods  presently  used  to  emulate  the  vidicon  camera  raster  scan) 
Present  commercial  devices  have  designed  the  imaging  array  output 
serially  only,  unfortunately.  A  parallel-processing  computer,  or  a 
hierarchy  of  serial-processing  computers  would  be  needed  to 
absorb  data  rapidly. 

In  certain  applications  pertaining  to  a  rapidly  changing  environ 
mem  (e  g  .  assembly  lines,  human  motion,  vision  aids  for  the 
blind.”  etc.),  all  the  necessary  coded  images  could  be  obumed  by 
using  a  duster  of  I  -♦  log,N  separate  active  cameras,  each  pro 
jectmg  a  single  beam  array 'pattern  at  a  different  laser  wavelength 
and  a  duster  of  I  -t  log,N  separate  selectively  filtered  passive 
cameras,  each  receiving  the  corresponding  pattern  reflections  in  the 
corresponding  wavelength.  Such  a  (I  r  log jNpfold  parallel  active- 
passive  camera  system  would  accomplish  data  collection  for  an  en 
tire  topography  in  one  cycle  time.  The  calibration  for  the  cluster 
cameras  would  be  only  slightly  more  complicated  than  for  the 
sequential  cameras 

Parallel  processing  of  dau  could  result  in  "almost  instan 
taneous"  scene  capture  and  analysis,  an  updated  3-D  map  could 
possibly  be  calculated  every  second,  or  faster.  A  two  millisecond 
scene  capture  with  one  to  ten  seconds  minicomputer  processing 
would  be  adequate  for  many  medical  and  industrial  applications.  A 
high-speed  synchronized  cinematic  process  could  be  used  as  an 
alternative  for  image  acquisition,  but  if  would  require  a  possibly 
unacceptable  postprocessing  film  development  delay 

One  ot  more  scene  capture  devices  (e.g  .  videocameras)  may  be 
placed  around  the  scene  which  is  being  illuminated  by  the  space 
coded  beam  arrays  Gimbal  mounts  may  be  used  for  these  cameras, 
as  well  as  various  robot  arm  mountings,  and  various  electro-optic 
and/oi  acoustic  aimers  can  be  used  to  keep  the  cameras  focused 
and  centered  on  the  scene,  even  if  the  camera(s)  and  beam  array 


projector  are  mobile  Calibration  and  positioning  requirements 
have  already  been  discussed  Camera-projector  angulations  can  be 
calculated  automatically  by  the  minicomputer  dedicated  to  the  3-D 
analysis.  Suitable  lens  systems,  determined  by  the  user  for  each  ap 
plication,  are  mounted  on  the  imaging  cameras 

User-defined  narrow -bandpass  optical  filters  foi  each  relevant 
User  wavelength  are  chosen  for  viewing  in  ambient  lighting  condi 
lions  If  eye  safely  constraints  require  low  levels  of  lasei  light  foi 
measuring  human  subjects,”  image  intensification  may  be  re 
quired  A  beam  array  pattern  illuminating  at  1  gW/cmJ  at  514  5  nm 
has  been  detected  reflecting  off  a  coarse  black  cardboard  when  ap 
propriately  selected  videocameras  or  externally  mounted  image  in¬ 
tensifies  were  used 

9.  further  data  processing  and  output 

Data  processing  is  required  for  (I)  hardware  control  to  synchronize 
the  programmable  laser-shutter  encoder  with  the  image  acquisition 
svstem  and  (2)  the  calculation  of  the  three-dimensional  number 
mapt  for  the  surface  topography  The  data  processor  may  also 
verify  the  accuracy  of  the  camera  orientation  vis-a-vis  the  laser  sur¬ 
face  target  and  warn  of  reflected  points  hidden  to  the  camera  by 
convolutions  in  the  surface  topography  Data  output  may  be  in  the 
form  of  digitally  stored  numerical  tables  Numerous  applications 
and  postprocessing  methods  are  possible  once  a  numerical  table  of 
the  3-D  coordinates  of  sufficient  sample  points  of  a  surface  is 
available  For  example,  graphic  display  output  and  numerical  con 
trol  of  machines  are  possible.  Serial  maps  may  be  rapidly  updated 
to  discern  movements  and  distortions  occurring  on  a  surface  being 
scanned  by  the  topographic  system  Straightforward  image  subtrac 
tion  techniques  using  sequentially  generated  output  maps  could 
lead  to  notification  and  3-D  vector  analysis,  in  real  time,  of  objects 
entering  or  moving  within  even  the  most  cluttered  scene  area 

10.  DISCUSSION 

The  objectives  of  this  effort  were  to  define  and  conceptually  create 
the  technological  capability:  (1)  to  obtain  unambiguously  and  from 
a  distance  the  three-dimensional  coordinates  of  a  large  number  of 
sample  points  of  any  open  uncovered  surface  of  any  object  (of  any 
size)  and  thereby  to  determine  the  surface  topography  of  the  ob 
jecl,  (2)  to  map  a  surface  rapidly,  remotely  ,  nondestructive^,  and 
automatically,  (3)  to  obtain,  where  possible,  spatial  data  from  as 
few  as  one  scene-viewing  direction,  (4)  to  generate  the  topographic 
data  in  real  time,  (5)  to  design  a  simple-to-operate.  self-contained, 
portable  system  whose  output  requires  no  human  processing  and 
whose  setup  requires  no  tedious  calibration  or  excessive  expertise. 
(6)  to  design  a  device  so  that  its  software  determines  insofar  as 
possible  the  geometric  transformations  among  the  sensor,  emitter . 
and  scene  (to  enable  decoding  of  the  image  data),  (7)  to  design  a 
system  for  maximum  versatility  and  ruggedness  fot  use  in  hazard 
ous  industrial,  clean  room  clinical  and  surgical,  and  military  en¬ 
vironmental  conditions,  (8)  lo  provide  human  oi  machine  interac¬ 
tive  capability  for  design  fabrication,  pattern  recognition  and  scene 
analysis  in  three  dimensions,  and  servo-controlled  robot-vision 
coupled  applications 

The  specific  concept  described  and  demonstrated  by  laboratory 
breadboarded  means,  and  outlined  mathematically,  has  certain 
uniquely  interesting  features  that  lend  themselves  to  industrial  anti 
medical  applications,  were  the  concept  developed  further 

(I)  The  same  apparatus  could  map  microscopic  or  large  size  ob 
jects  merely  by  choosing  suitable  end  objective  lenses  The  beam  at 
ray  is  composed  of  discrete  beams  at  infinite  focus,  so  that  an  ob 
ject  intercepting  a  beam  need  not  be  at  a  predetermined  distance 
from  the  origin  of  the  array.  Pattern  array  generation  is  the  product 
of  diffraction,  so  that  the  array  may  be  enlarged  ot  reduced  at  will 
or  projected  a  long  distance,  avoiding  the  diffraction-limiting  ef 
feels  that  occur  when  "grids"  are  produced  with  bmitmg  aperture: 
or  masking  slits.  No  moving  parts  or  mechanical  devices  are  used 
for  array  generation  or  for  space  coding,  and  the  electro-optic  can- 
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didate  materials  appear  lo  have  long  duly  cycles,  possibly  permit- 
ling  the  active  camera  optics  to  be  sealed  for  maintenance-free  use. 
With  reasonable  selection  of  optics,  the  array  generation  would 
work  with  most  User  wavelengths  or  power  outputs. 

(2)  The  active  (projector)  and  passive  camera(s)  can  be  posi¬ 
tioned  (moved)  independently  at  arbitrary  axes  (e.g.,  on  indepen¬ 
dent  robot  arms)  for  observation  of  the  scene  from  different 
perspective  directions  after  each  set  of  data  is  acquired  Thus  a 
fixed  rotation  axis  for  the  camera  system  and/or  the  part  to  be  in¬ 
spected  is  not  necessary 

(3)  The  active  illumination  of  medical  or  industrial  pans 
eliminates  ambiguities  obtained  with  passive  systems  due  to  (a) 
natural  protective  coloration,  and  (b)  variations  in  reflective  prop¬ 
erties  as  a  result  of  mixed  textures,  machine  surface  finishings,  dif¬ 
fering  materials,  coatings,  and  clothings.  Active  beam  illumination 
creates  discrete  peak  reflections  (even  if  of  variable  amplitude)  on 
the  object  surface,  and  a  known  finite  array  of  ordered  beams  can 
be  accounted  for  as  either  present  or  missing 

(4)  Any  suitable  recording  sensor(s)  could  be  used  for  the  passive 
camera(s).  A  videocamera  ouipul  by  whatever  means  could  be 
rapidly  converted  to  a  standard  TV  broadcast  system  for  remote 
transmission  of  image  data  in  very  near  real  time,  permitting 
remote  monitoring  of  automated  system  3-D  measurement  With 
lasers  of  different  wavelengths  and  correspondingly  filtered 
vtdeocamcras.  more  than  one  beam  array  pattern  may  be  projected 
and  simultaneously  and  independently  viewed  Thus  simultaneous 
coarse/fine  scanning  and  global/regional  scanning  of  a  Urge  object 
or  parts  (for  positioning  or  comparative  dimensional  analysis)  is 
possible 

(5)  Gross  deformations  may  be  comparatively  and  accurately 
measured  against  a  data  base.  Fully-3-D  deformation  analysis  in 
.gross  increments  is  especially  useful  for  comparison  of  rough 
castings  or  forgings,  measurements  of  bubble  expansion,  and 
choosing  which  of  a  set  of  similar  Urge  industrial  assemblies  should 
be  mated  to  another  with  the  least  hand  fitting 
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APPENDIX  D 

LASER  SCANNER  CONCEPT 


The  following  pages  outline  a  laser  scanner  system  (Figure  D-l)  and  its 
operational  parameters.  Although  this  particular  system  was  designed  by  ERIM, 
similar  systems  have  been  assembled  and  tested  by  JPL,  Perkin-Elmer,  and 
others.  Implementation  details  vary,  but  all  have  essentially  the  same  basic 
configuration. 

Positive  attributes  of  this  concept  include: 

•  no  lost  data  due  to  obscuration  or  shadowing, 

•  computational  and  image  storage  requirements  are  minimized, 

•  in  the  CW  mcde,  reflectance  data  are  also  obtainable  for  imaging,  and 

•  system  may  be  directed  to  interrogate  one  point  of  interest  without 
interrogating  the  entire  field. 

Areas  which  could  use  additional  work  include: 

•  higher  power  laser  sources,  particularly  semiconductor, 

•  more  efficient,  nonmechanical  means  to  achieve  beam  steering, 

•  inherent  range  ambiguity  in  CW  mode,  and 

•  data  acquisition  rates  in  general. 
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Figure  D-l.  Table  Top  Three-Dimensional  Scanner  (T3DS) 


Table-Top  3-D  Scanner  Specifications 
SOURCE 


Mode-Locked  He-Ne  Laser,  Spectra  Physics  Model  125 


Power: 

Beam  Size: 

Beam  Divergence: 

Basic  Pulse  Frequency: 
Pulse  Width: 

Harmonics  Used: 

Range  Ambiguity: 


=50  mW  maximum  (~7  mW  needed) 
2  mm 
=1  mr 
=79. 1  MHz 
1  ns 

Fund,  3  and  6 

189.5  cm,  63.2  cm  and  31.6  cm 


RECEIVER 


Photomultiplier  ITT  Type  4100,  frequency  response  to  500  MHz 


Collecting  Aperture: 
RF  Electronics: 

Phase  Detector: 

RMS  Range  Accuracy: 

SCANNING  PARAMETERS 


10  mm  diameter  lens 

Commercial  filters,  mixers,  amplifiers 
Analog  Pulse  Type  (from  AVD-4  electronics) 
-3mm  for  all  range  ambiguities 


Maximum  Scanning  Field  of  View:  14°  x  14°  or  480  mm  x  480  mm  at  2-m  range 
IF0V  (spot  size):  2  mm  -  10  mm  at  3-m  range 

Frame  Rate:  1/20  seconds  limited  by  recording 

Pixels/Line:  128,  256,  or  512 
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APPENDIX  E 

ULTRASONIC  PHASE  MONITORING 


Using  ultrasonics  as  a  sensing  technology  has  both  positive  and  negative 
aspects.  A  properly  configured  system  can  be  sensitive  to  extremely  small 
dimensional  changes,  but  the  system  generally  has  a  very  limited  ability  to 
localize  or  identify  the  source  of  those  changes.  Thus  in  a  monitoring  or 
inspection  role,  ultrasonics  can  be  very  Important,  but  it  has  limited 
applicability  in  a  generalized  imaging  role. 

The  following  paper*  discusses  the  use  of  ultrasonics  in  an  inspection 
and/or  monitoring  role.  Implied  sensitivities  are  excellent,  providing 
certain  configuration  constraints  are  met.  In  these  roles,  and  in  proximity 
sensing,  ultrasonics  will  undoubtedly  be  a  major  contributor  to  the  growth  of 
robotic  technology. 


♦Reprinted  from  Transactions  of  the  Society  of  Manufacturing  Engineering, 
Volume  6,  1978. 
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PHASE  MONITORING  FOR  AUTOMATED  INSPECTION,  POSITIONING  AND  ASSEMBLY 

by 

SHAWN  BUCKLEY,  Associate  Professor,  Massachusetts  Institute  of  Technology 
KIM  STELSON,  NSF  fellow,  Massachusetts  Institute  of  Technology 

ABSTRACT.  Phase  sxinitoring  is  a  technique  which  can  be  used  in  several  aspects  of  automated  manu¬ 
facturing:  inspection,  assembly  and  positioning.  In  phase  Monitored  inspection,  acoustic  waves 

reflected  off  a  Master  part  are  picked  up  by  an  array  of  Microphones.  The  phase  shift  of  the  re¬ 
flected  wave  in  each  microphone  is  compared  to  the  subsequently  measured  values  of  a  part  to  be 
inspected;  differences  in  these  phase  signals  can  be  interpreted  as  geometry  differences  between 
master  and  part.  Similarly,  the  phase  aenitoring  technique  can  be  used  for  automated  assembly  by 
attaching  the  sound  source  and  microphone  array  to  a  manipulator  hand;  the  phase  signals  determine 
the  relative  position  between  the  manipulator  and  the  object  being  assembled.  Automated  position¬ 
ing,  on  the  other  hand,  uses  sisiilar  phase  signals  to  position  objects  in  the  proper  orientation 
before  further  Machining  or  assembly. 

Phase  Monitoring  is  suitable  for  low  production  runs  (where  economics  eliminate  specialized 
transfer  line  equipment)  since  it  can  be  easily  recalibrated  for  different  part  shapes.  PM  is  ex¬ 
pected  to  be  low  cost — estimated  at  between  S2.000  and  $5,000  per  system — because  its  components 
are  all  off-the-shelf  speakers,  microphones  and  microprocessors.  Other  advantages  are  high  speed 
(several  hundred  measurements  a  second),  no  moving  parts,  and  reasonable  stand-off  distance  (sev¬ 
eral  inches).  Accuracy  is  high:  changes  in  geometry  of  as  small  as  a  mil  (  03mm)  can  be  detected 
with  present  equipment. 


INTRODUCTION .  Inspection  and  assembly  today 
are  the  least  automated  of  manufacturing  oper¬ 
ations,  though  some  automation  of  these  opera¬ 
tions  does  exist. 1  Automated  inspection  can 
be  as  sophisticated  as  Bendix's  Cordax  inspec¬ 
tion  machine  operating  under  computer  control;^ 
automated  assembly  can  be  as  sophisticated  as 
Kawasaki's  assembly  of  motorcycle  engines. 
However,  most  shop  practice  is  with  hand-oper¬ 
ated  inspection  and  assembly  equipment:  very 
little  automation  has  worked  its  way  into  these 
processes. 

Consider  automated  inspection  as  an  illus¬ 
tration.  It  has  been  applied  most  successfully 
to  large  production  runs.  For  example  in  the 
Manufacture  of  automobile  connecting  rods, 
precision  inspection  of  the  machined  surfaces 
is  automatic.  Because  of  the  large  number  of 
nearly  identical  parts,  economics  dictate  an 
inspection  ’transfer  line’.  The  inspection 
Machine  is  specially  designed  for  each  part 
ahape.  While  effective  for  large  production 
runs  (a  million  parts  or  more)  the  transfer 
line  methods  are  not  economical  for  low-volume 
inspection.  An  inspection  system  Must  be  flex¬ 
ible  to  inspect  parts  of  many  different  shapes. 

Phase  monitoring  is  a  technique  which  can 
eliminate  much  of  the  labor  involved  in  hand 
inspecting  and  assembly  for  the  smaller  pro¬ 
duction  runs  where  transfer  line  equipment  is 
not  suitable.  Phase  monitoring  (PM)  is  flexi¬ 
ble  enough  to  be  easily  recalibrated  for  each 
new  part  shape.  It  is  expected  to  be  a  low- 
cost  method — estimated  costs  are  between  $2,000 
and  $5,000  per  inspection  station.  PM  tech¬ 
niques  have  other  advantages  of  high  speed 
(several  hundred  measurements  a  second) ,  no 
moving  parts  and  reasonable  stand  off  distances 
(several  inches) . 

Automated  inspection  will  be  used  to  illus¬ 
trate  the  PM  technique;  assembly  and  position¬ 
ing  will  be  shown  later  to  be  variations  of 
inspection.  Figure  1  shows  a  schematic  of  a 
PM  automated  inspection  system.  An  array  of 
wave  emitters  and  wave  receivers  are  located 
around  the  object  to  be  inspected:  for  simpli¬ 
city,  only  a  single  emitter  is  shown.  The 
emitter  sends  out  acoustic  waves  of  a  constant 
wavelength  which  arc  reflected  from  the  object 
and  then  picked  up  by  the  receivers.  As  a  re¬ 
sult  of  the  constant  wavelength  emission,  the 
output  of  the  receivers  will  be  a  sinusoidal 
signal  of  the  tame  frequency  as  the  emitted 


wave  but  differing  in  both  amplitude  and  phase. 
Inspection  of  the  part  is  made  possible  by  moni¬ 
toring  the  phase  differences  between  emitted 
and  received  waves. 

First,  a  master  workpiece  is  positioned  at 
the  inspecting  station.  The  plurality  of  the 
phase  measurements  monitored  at  each  receiver 
constitutes  the  ’phase  vector"  of  the  aiaster 
workpiece: 

•  “  (ey.Oj . »„)  -  (1) 

A  subsequent  object  to  be  inspected  is  po¬ 
sitioned  at  the  same  point  and  orientation,  and 
a  similar  phase  vector: 

e'  -  . •'„>  ,2> 

is  monitored  for  the  object.  The  difference  in 
phase  between  the  master  and  the  object: 

AS  «  (Sj-S'j.Sj-S'j . en“BV  ,3) 

can  be  used  to  determine  whether  a  part  is  with¬ 
in  tolerance,  where  it  isn't,  and  by  how  much. 
Assume  for  the  time  being  that  the  object  and 
master  can  be  placed  in  the  aame  orientation  and 
position;  this  positioning  can  be  either  by  con¬ 
ventional  methods  or  by  PM  positioning  (to  be 
discussed  later) . 

For  many  gross  errors  the  object  can  be  re¬ 
jected  if  the  phase  difference  vector  is  simply 
unrecognizable.  Smaller  errors  require  more 
sophistication.  A  sensitivity  matrix,  S,  can 
be  determined  which  gives  the  sensitivity  of 
each  phase  difference,  asj  ,  for  each  dimension 
tolerance  of  the  object;  each  element  of  S  is 
the  partial  derivative  of  a  particular  dimen¬ 
sion  to  the  phase  change  at  a  particular  micro¬ 
phone.  Multiplication  of  the  phase  difference  ° 
vector  times  the  inverse  of  the  sensitivity  ma¬ 
trix  results  in  a  deviation  vector:  * 

d  -  S"1**  (4) 

The  deviation  vector  is  the  amount  that 
each  dimension  of  the  object  has  changed  or 
•deviated"  from  that  of  the  master  workpiece: 

d  -  (dj.dj . d„>  .  (5) 

If  any  change  is  outside  the  tolerance  for 
that  dimension  the  object  is  rejected. 
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FIGURE  1  SCHEftATIC  OF  Pfl  mPECUOH 


To  have  sufficient  information,  the  num¬ 
ber  of  phases  measured  must  be  greater  than  or 
equal  to  the  number  of  dimensions  inspected. 
Redundant  phase  measurements  are  likely  to 
provide  more  reliable  estimates  of  d.  Hoie- 
over,  they  alao  mitigate  the  possibility  that 
S  may  be  singular  or  nearly  singular.  In 
short,  more  phase  measurements  increase  the 
probability  that  every  dimension  is  well  mea¬ 
sured. 

Nhen  the  nuir  of  measurements  exceeds 
the  number  of  dim  signs,  S  will  be  .ion-square 
and  the  inverse  of  S  docs  not  exist.  A  pseudo- 
inverse  must  be  used. 3  The  pseudo-inverse  of 
S  is  not  unique.  The  residual,  r,  is  given  by: 

r  «  S  d-AS  (6) 

The  pseudo-inverse  that  minimized  the 
Euclidean  norm  of  the  residual,  r,  is  called 
the  left  minimum  inverse.  Since  minimizing 
the  norm  of  the  residual  is  the  same  as  mini¬ 
mizing  the  sum  of  the  squares  of  the  components 
of  r,  the  left  minimum  inverse  is  simply  a 
least-squares  fit  of  the  dimensions  to  the  ex¬ 
cess  measurements.  The  left  minimum  inverse 
is  given  by: 

S  E  (sTS)  1  ST.  (7) 

Then,  the  deviation  vector  can  be  com¬ 
puted  as  follows: 

d  -  S^AS  («) 


The  sensitivity  matrix  described  above 
assumes  linear  changes  in  phase  for  small  de¬ 
viations.  This  will  be  true  if  the  emitted 
wavelength  Is  much  greater  than  the  largest 
possible  deviation.  A  one-inch  (25mm)  wave¬ 
length  allows  deviations  as  large  as  s.050  inch 
(1.3mm)  to  be  measured  while  still  satisfying 
this  criterion.  A  one-inch  125mm)  wavelength 
corresponds  to  a  frequency  of  about  10  KHz  in 
air-well  within  the  realm  of  hi-fi  equipment. 
However,  a  slightly  higher  frequency  le.g., 

20  to  30  KHz)  might  be  convenient  to  keep  the 
emitted  waves  Out  of  the  audible  range. 

\ 

The  sensitivity  matrix  itself  is  determined 
by  a  calibration  procedure  performed  only  once 
for  each  master  workpiece.  It  involves  matching 
the  phase  differences,  f.e,  with  actual  devia¬ 
tions,  d,  of  objects  with  high  and  low  toler¬ 
ance  limits.  Once  the  sensitivity  matrix  has 
been  determined,  all  subsequent  objects  charac¬ 
terized  by  that  particular  master  workpiece  can 
be  inspected  on-line,  without  the  need  to  con¬ 
tinually  recalibrate.4 

As  you  have  seen,  by  paying  special  atten¬ 
tion  to  phase  changes  of  reflected  acoustic 
waves,  the  deviation  of  a  part  from  the  master 
can  be  determined.  PM  assembly  and  PM  position¬ 
ing  use  similar  methods  of  matrix  manipulation. 
The  simplest  case  is  PM  positioning  where  an 
object  (e.g.,  a  transmission  casting)  is  posi¬ 
tioned  on  a  pallet.  Automated  positioning  is 
useful  in  certain  automated  manufacturing  pro-  - 
cesses  when  a  part  must  be  located  in  some  known 
position  for  subsequent  machining  or  assembly 
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operations ■  Positioning  is  a  simple  task  be* 
cause  at  most  there  are  *ix  possible  devia¬ 
tions  correspond ing  to  three  degrees  of  trans¬ 
lation  and  three  degrees  of  rotation.  However, 
in  an  actual  positioning  operation,  often  only 
two  or  three  degrees  are  required  (rotation 
and  XV  positioning).  In  PM  positioning,  a  de¬ 
viation  is  defined  as  the  error  from  a  nosunal 
position  rather  than  a  change  in  dimension  as 
in  PM  inspection.  So  three  degrees  of  posi¬ 
tioning  constitute  an  S  matrix  with  only  three 
deviations.  Xs  long  as  the  deviations  are 
relatively  small  compared  to  the  wavelength 
(aay,  1/8  inch  (3.2mm)  compared  to  an  inch 
(25«*o)  wavelength)  the  same  theory  applies  to 
PM  positioning  as  PM  inspecting. 

PM  assembly  can  be  thought  of  as  many  con¬ 
tinuous  PM  positioning  steps  in  succession. 
Imagine  an  array  of  microphones  and  an  acoustic 
emitter  attached  to  a  manipulator  hand.  In 
each  position  of  the  hand,  the  phase  signals 
will  represent  the  relative  position  of  ob¬ 
jects  close  to  the  hand.  Thus  the  hand  can 
follow  some  predetermined  trajectory  bated  not 
on  absolute  XY2  coordinates  but  rather  on  co¬ 
ordinates  associated  with  the  objects  being 
-assembled.  Since  most  current  manipulators 
can  easily  arrive  within  an  eighth-inch  (3.2sm) 
of  some  predetermined  position,  PM  assembly 
becomes  a  vernier  calibration  movement,  re¬ 
positioning  the  manipulator  hand  relative  to 
the  objects  being  assembled.  For  this  appli¬ 
cation  phase  monitoring  is  accurate  to  a  mil 
(,03mm),  allowing  accurate  repositioning  of 
the  manipulator '  s  coordinates.  Furthermore, 
the  PM  system  need  not  necessarily  be  on  the 
manipulator  hand--lt  could  be  integrated  into 
the  holding  fixtures  of  the  assembly  system. 
Eince  the  PM  technique  measures  relative  po- 
sitions,  it  makes  no  difference  whether  it  is 
on  the  fixture  "seeing"  the  hand. 

Phase  monitoring,  then,  is  a  way  of  find¬ 
ing  "deviations"  in  objects.  The  deviations 
can  either  be  geometry  changes  from  object  to 
object  (inspection),  changes  in  a  single  ob¬ 
ject's  position  (positioning)  or  changes  in 
the  relative  positions  between  objects  (assem¬ 
bly)  .  The  use  of  phase  as  a  measurement  vari¬ 
able  has  inherent  advantages.  By  analogy, 
consider  a  tape  measure  which  measures  distance 
giving  only  fractions  of  an  inch,  not  the  num¬ 
ber  of  inches.  However,  this  tape  measure  can 
measure  these  fractions  very  accurately,  aay, 
to  one  part  in  a  thousand.  Suppose  you  were 
to  use  this  tape  Mature  to  find  the  length 
of  an  object  whose  nominal  measurement  was  10 
inches  (.4m) .  If  you  measured  .001  inches 
( . 03mm)  you  would  know  it’s  relative  tolerance 
to  one  part  in  ten  thousand,  even  though  your 
measurement  vus  only  accurate  to  one  part  in  a' 
thousand.  Similarly,  phase  can  be  steasured  to 
one  part  in  a  thousand  even  though  the  distance 
between  object  and  the  microphone  array  say  be 
several  wavelength.  In  most  practical  circum¬ 
stances  measuring  the  proper  phase  but  the 
wrong  wavelength  (9.001  instead  of  10.001  in 
the  tape  measure  analogy)  won't  occur  if  the 
phase  changes  are  small  compared  to  a  wsve- 
length.  In  inspection,  positioning  and  assem¬ 
bly,  all  changes  are  small  giving  advantages 
to  the  inherent  sensitivity  of  phase  measure¬ 
ment. 


to  allow  accurate  movement  in  two  orthogonal 
directions.  Micrometer  heads  on  the  transla¬ 
tion  stages  allowed  the  carburetor  to  be  posi¬ 
tioned  accurately. 

Koving  the  carburetor  slightly  in  each  of 
two  directions  (ax, ay)  caused  phase  changes  in 
each  of  four  microphones  (atj.aSj.aa j.ao^,) . 
The  resulting  sensitivity  matrix  was: 


General  Experimental 

Expression  Data 


Each  element  of  the  S  matrix  is,  in  essence, 
the  partial  derivative  of”each  microphone's 
phase  difference  with  respect  to  each  deviation. 
Given  the  sensitivity  matrix,  any  small  posi¬ 
tion  change  of  the  carburetor  in  the  x  or  y  di¬ 
rection  could  be  found  by  monitoring  the  phase 
differences  at  each  microphone. 

To  illustrate  the  matrix  technique,  con¬ 
sider  the  case  where  the  carburetor  was  moved 
10  xtils  (.25nm>)  in  both  the  x  and  y  directions. 
The  Masured  phase  change  vector  for  this  com¬ 
bined  movement  was: 


degrees  (10) 


To  exasdne  the  improvement  in  accuracy 
achieved  by  using  redundant  phase  measurements, 
comparisons  were  made  among  three  different 
cases:  1)  using  only  the  first  two  microphones, 
2)  using  only  the  last  two  microphones  and  3) 
using  all  four  aucrophones.  The  first  two 
cases  involve  the  inversion  of  a  square  sensi¬ 
tivity  matrix;  the  last  case  involves  pseudo¬ 
inversion  of  a  non-square  sensitivity  matrix. 


Using  only  the  first  two  microphones  (i.e., 
the  top  four  elements  of  Equation  9  and  the  top 
two  elements  in  Equation  10) : 


S  - 


-0.645 

-0.590 


-0.555 

1.045 


degrees 

mil 


(11) 
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Since  the  actual  movement  of  the  carburet* 
or  was: 

d" 
x 

d 

.  y. 

using  only  the  firat  two  microphone*  did  not 
provide  a  very  good  estimate  of  the  carburetor 
movement . 

Similarly,  using  only  the  last  two  micro¬ 
phones  (the  bottom  four  elements  of  Equation  9 
and  the  bottom  two  elements  in  Equation  10), 
gave  a  poor  estimate  of  the  actual  movement. 
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However,  using  all  four  microphones  (all 
the  elements  of  Equations  9  and  10)  gives: 
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For  this  case,  pseudo-inversion  provides  a 
substantial  improvement  in  the  estimate  of  the 
deviation  from  the  actual  value  of  10  mils 
(.2Smm)  in  each  direction.  This  will  not  be 
universally  true:  the  pseudo-inverse  provides 
only  a  most  likely  estimate  of  the  actual  de¬ 
viations.  If  the  point  picked  for  calculation 
were  near  a  relative  maxima  or  minima  in  the 
phase  variation,  higher  order  effects  might  be 
more  influential,  causing  decreased  accuracy. 
This  possibility  was  avoided  for  the  purpose 
of  the  example  shown. 


In  general,  geometry  changes  or  deviations 
can  be  divided  into  two  sixes:  1)  changes  in 
objects  whose  surface  area  is  greater  than  a 
wavelength  squsred  and  2)  changes  in  objects 
whose  surface  area  is  smaller  than  a  wave¬ 
length  squsred.  Recall  the  PH  systems  use  a 
wavelength  about  an  inch  (2Srun)  long,  so  a  wave¬ 
length  sousred  has  about  one  square  inch 
(6  x  10'4m2)  surface  area.  For  big  surfsces, 
the  rsy-trscing  method  has  been  found  to  give 
the  most  insight  into  applications  of  phase 
monitoring.  PH  positioning  and  PH  assembly 
usually  involve  these  large  surfaces.  In  as¬ 
sembly,  the  'surfaces*  are  the  objects  and  fix¬ 
tures  'seen*  by  the  microphone  srray  in  the 
manipulator  hand.  If  the  array  were  mounted  in 
the  holding  fixture,  the  'surfaces*  would  be 
those  of  the  manipulator  hand  itself  as  it  ap¬ 
proached  the  objects  being  sssembled.  In  PH 
positioning  the  'surfaces*  are  just  those  of 
the  object  being  positioned.  In  both  PH  assem¬ 
bly  and  PH  positioning  small  relative  movement 
of  objects  with  large  surface  area  is  the  impor¬ 
tant  theme. 

PH  inspection,  on  the  other  hand,  usually 
deals  with  geometry  changes  of  small  surfaces. 
Huygen's  diffraction  theory  must  be  used  for 
analysis  since  the  ray-tracing  method  is  not 
appropriate.  For  example,  detecting  the  pres¬ 
ence  or  absence  of  a  machine  screw  on  an 
assembled  part  must  use  diffraction  theory  be¬ 
cause  the  surface  area  of  the  machine  screw  is 
So  small. 

In  analy2ing  large  surfares,  the  ray-trac¬ 
ing  method  assumes  three  types  of  acoustic 
‘rays*  are  combined  in  each  microphone:  the 
direct  ray,  reflected  rays  and  diffracted  rays. 
The  direct  ray  is  the  pressure  signal  coming 
directly  from  the  wave  emitter  to  a  particular 
microphone.  Reflected  rays  reach  the  micro¬ 
phone  by  being  reflected  from  the  object's  sur¬ 
face;  geometric  reflection  is  assumed.  Dif¬ 
fracted  rays  are  emitted  from  each  sharp  corner. 
Figure  2  shows  direct,  reflected  and  diffracted 
rays  converging  on  a  microphone  in  s  very  simple 
geometry:  a  two  dimensional  wedge.  These  waves 
are  actually  time-varying  pressure  wsves  of 
different  amplitude  and  phsse  shift  which  can 
be  combined  vectrially  using  phasor  diagrams 
(see  Fioure  2) . 


In  PH  inspection  and  PH  assembly,  the 
ssme  matrix  techniques  are  valid.  As  mentioned 
before,  in  PH  inspection  the  phsse  signals  of 
s  part  are  compared  to  a  master.  In  PH  assem¬ 
bly  the  phsse  signals  correspond  to  relative 
positions  between  the  objects  being  ssseabled. 
However,  the  distinction  between  the  vsrious 
PH  techniques  is  blurred.  For  example,  PH  in¬ 
spection  requires  that  subsequent  parts  be 
placed  in  exactly  the  same  position  so  that 
phase  differences  correspond  only  to  differences' 
between  part  and  master.  However,  it's  pos¬ 
sible  to  optimally  position  the  part  using  PH 
positioning  and  then  subsequently  PH  inspect  it. 
In  fact,  if  there  are  only  small  errors  in  its 
position,  the  part  can  be  inspected  without 
repositioning:  the  matrix  technique  esn  *wssh 
out*  the  effect  of  misalignment. 

RESOLUTION.  While  the  previous  discussion  has 
shown  how  phase  monitoring  using  several  micro¬ 
phones  can  measure  geometry  changes,  the  accu¬ 
racy  associated  with  the  measurement  hss  not 
been  given.  How  small  a  defect  can  be  detected?. 
Can  microphones  be  placed  in  optimal  positions 
to  detect  tolerances  best?  These  and  similsr  ' 
questions  can  be  answered  by  turning  to  acous¬ 
tic  theory. 


To  investigate  the  validity  of  the  rsy- 
trscing  technique,  one  of  the  authors  (Stelson) 
hss  studied  rsy-trscing  experimentally.  He 
used  a  two-dimensional  wedge  geometry  as  shown 
in  7.  An  omni-directional  source  and 

microphone  were  held  fixed  while  the  wedge  was 
moved  in  two  dimensions,  simulating  PH  position¬ 
ing  of  a  simple  object.  The  results  of  the 
experiment  are  shown  in  Figure  3.  The  vertical 
scale  shows  absolute  phase  shift  as  received  by 
the  microphone  while  the  horizontal  axis  shows 
the  wedge's  position.  A  theoretical  prediction 
of  the  phase  at  the  microphone  is  shown  along 
with  experimental  data.  The  prediction  used  s 
computer  program  that  computes  the  combined 
effects  of  direct,  reflected  and  diffrseted 
rsys.  The  theory  agrees  well  with  the  experi¬ 
ment,  verifying  ray-tracing  for  determining 
sbsolute  phase.  The  errors  between  theory  and 
experiment  in  Figure  3  are  not  indicative  of 
the  precision  with  which  PH  can  measure  changes 
in  geometry;  they  represent  the  differences  be¬ 
tween  an  ideal  wedge  and  the  real  wedge.  Note 
that  the  data  presented  is  merely  representa¬ 
tive;  other  tests  shewed  similsr  correlation 
with  theory.5 
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Pot  objects  with  small  surface  area,  a 
variation  of  Huygen'a  diffraction  theory  la 
appropriate.  In  this  theory,  pressure  waves 
striking  an  area-increment  much  smaller  than 
the  wavelength  squared  are  assumed  to  disperse 
omni-directionally  awsy  from  the  plane  of  the 
surface.  The  pressure  signal  from  the  surface 
becomes  the  phasor  combination  of  signals  from 
each  area-increment .  Each  area-increment  be¬ 
haves  as  if  it  were  a  small  pressure  source 
located  on  the  object's  surface.  The  phasor 
sum  of  these  sources  is  coafeined  with  the  di¬ 
rect  pressure  aignsl  coming  from  the  sdcro- 
phone,  as  well  as  with  reflected  signals  coming 
from  other  nearby  objects. 

The  diffrsetion  theory  wss  experimentally 
verified  by  the  senior  author.  Cylinders  were 
pushed  through  close-fitting  boles  in  a  flat 
surface  to  simulate  the  effect  of  a  small  de¬ 
fect  in  the  surface.  The  cylinders  began 
flush  with  the  surface  and  were  raised  an 
eighth  inch  (3.2mm)  above  the  surface  as  shown 
in  Plgure  4.  The  phase  of  the  microphone  aig- 
nal  was  primarily  determined  by  direct  wsves 
from  the  emitter  and  to  a  lesser  degree  by 
reflected  wave*  from  the  fist  surface.  This 
dominant  signal  was  changed  slightly  by  the 
contribution  of  the  signal  diffracted  from  the 
"defect*. 

Results  of  the  ssiall-surfsce  experiment 
are  shown  in  Figure  5 .  The  vertical  axis 
•hows  the  phase  change  (the  difference  in  the 
microphone  signal  before  and  after  the  cylinder 
movement)  the  horizontal  acale  shows  the  diam¬ 
eter  of  the  cylinder.  Also  shown  in  Plgure  i 
are  the  results  of  a  computer  program  using 
the  Huygen'a  diffraction  principle.  Since 
the  exact  phase  chsnge  depends  on  the  ration 
the  direct  and  reflected  components  compared 
to  the  diffracted  component  (which  is  very 
difficult  to  determine  analytically),  the 
theoretical  results  have  been  normalised  to 
the  experimental  results  toy  the  half-inch 
(12. Ton)  diameter  cylinder. 


Diffraction  theory  predicts  the  phase 
change  due  to  displacement  fer  each  tiny  area- 
increment  of  the  cylinder's  face  will  give  ap- 
proximately  the  same  phase  signal  change  at  the 
microphone.  Since  a  half-inch  (12.7mm)  diameter 
cylinder  has  four  tiaws  as  many  area- increments 
as  a  quarter-inch  ((.Sim)  diameter  cylinder, 
one  would  expect  its  phase  signal  to  change 
•bout  four  tines  as  much.  The  experiment  veri¬ 
fies  this  prediction.  In  general,  the  phase 
Change  is  proportional  to  the  area  of  a  small 
surface  being  inspected. 

What  do  these  experliacnts  mean  in  terns  of 
a  real  inspection,  positioning  or  assembly  task? 
Our  present  equipment  can  detect  phase  changes 
of  about  .001  cycles  (.006  rad).  This  swans 
for  big  surfaces  (ususlly  a  positioning  or 
asseaibly  task),  an  object  can  be  located  to  with¬ 
in  a  mil  (.03mn)  using  a  one-inch  (23mm)  wave¬ 
length.  For  small  surfaces  (ususlly  inspection 
tasks),  the  concept  of  a  "defect  volume"  is 
appropriate;  it  is  simply  the  volume  of  a  small 
defect  in  a  part  being  inspected.  Our  present 
equipment  can  detect  defect  volumes  as  small  as 
.002  cubic  Inches  (30mm3) .  Thus  s  half-inch 
(12.7sm)  diameter  defect  only  10  mils  (.2Sam) 
high  esn  be  detected,  but  an  eighth-inch (3. 2am) 
dlaswter  defect  must  be  an  eighth  of  an  inch 
(3.2am)  high  to  be  detected.  Diffraction  sees 
bumps  and  holes  as  the  ssme  so  the  resolution 
volume  applies  to  "positive"  and  "negative"  de¬ 
fects.  Mote  that  these  resolutions— 1  mil 
(.03mm)  positioning  for  lsrge  surf sees  and 
.002  in*  (30amJ)  defect  volumes  for  small  sur¬ 
faces— only  applies  to  our  present  inch-long 
(2Sam)  wavelength.  By  using  higher  frequency 
sound  greater  resolution  is  possible,  but  at 
the  expense  of  smaller  standoff  distance. 

COMPARISON  WITH  OTHER  SYSTEMS.  Two  types  of 
syatesis  are  currently  iwing  used  for  automated 
inspection,  positioning  and  assembly.  They  can 
loosely  be  referred  to  as  "point-to-point* 
techniques  and  "video"  techniques. 
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FIGURE  5  RESULTS  OF  DIFFRACTION  EXPERIMENT 


Point-to-point  machines,  used  primarily 
(or  inspection,  are  laden  with  aiechanical  or 
electronic  sensors  which  probe  the  part  for 
•ny  anomalies.  The  sensors  are  mechanically 
moved  into  close  proximity  of  the  surface  be¬ 
ing  inspected,  since  the  sensors  require  small 
standoff  distances.  Motion  is  controlled  by 
servo-motors  or  stepper  motors  similar  to  the 
motion  in  an  N/C  machine  tools;  however,  a 
sensor  replaces  the  cutter.  As  their  name  im¬ 
plies,  point-to-point  machines  make  ateasure- 
aents  at  individual  points  one  after  the  other. 

On  the  other  hand  video  techniques  gener¬ 
ally  use  eithci  linear  or  area  arrays  of  light 
detectors.  An  ordinary  television  casters 
would  fall  into  this  cateqory  (although  most 
recent  devices  use  solid-state  detectors). 
Coupled  with  a  microprocessor,  video  techniques 
have  automated  both  inspection  and  asseatbly 
processes. 

Compared  to  point-to-point  machines,  PM 
is  both  faster  and  has  averaging  capability. 
Since  the  point-to-point  machines  measure  se¬ 
quentially  taking  perhaps  a  second  for  each 
point,  a  part  with  many  measurements  can  take 
a  minute  or  more  for  inspection.  A  PM  aystca 
makes  all  its  measurements  in  parallel,  rather 
than  in  series,  end  so  is  inherently  faster. 


A  PM  inspection  machine  could  measure  10  or  20 
dimensions  in  less  than  a  second.  A  point-to- 
point  machine  is  only  sensitive  to  the  actual 
point  being  measured  (it  hss  no  way  of  knowing 
if  a  defect  exists  at  other  than  the  measured 
point).  The  capability  of  averaging  many  mea¬ 
surements  over  an  entire  surface  is  often  nec¬ 
essary  when  working  with  objects  such  as  aand 
castings.  Although  the  surface  of  a  casting  has 
a  rough  surface  with  many  flaws,  the  PM  technique 
can  detect  the  nominal  location  of  the  surface. 

Video  techniques  are  just  as  fast  as  PM, 
but  they  seem  to  be  limited  to  two-dimensional 
shapes.  The  variable  surface  reflectance  of 
three-dimensional  objects  often  gives  spurious 
results  to  video  techniques;  when  an  object  can 
be  presented  as  a  silhouette,  video  is  at  its 
finest.  TM  systems  inherently  use  three-dimen¬ 
sional  information  of  objects.  Furthermore,  in 
transforming  the  acoustic  waves  into  phase  sig¬ 
nals,  the  shape  of  the  object  becomes  unimpor¬ 
tant;  turbine  blades  are  no  more  difficult  to 
measure  than  spheres  or  cubes.  Although  both 
PM  and  video  use  microprocessors,  PM  has  the 
advantage  of  much  smaller  microprocessor  memory 
sise  and  less  computation  time.  Whereas  video 
techniques  store  and  stipulate  information 
associated  with  a  picture  of  the  entire  object, 
in  PM  only  the  barest  minimum  of  information  is 
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stored  or  swnipulated.  For  example,  using  10 
microphones  means  only  ten  10-bit  words  need 
be  stored  and  manipulated. 

Economics  are  another  way  to  compare  the 
various  systems.  A  point-to-point  inspection 
machine  is  usually  in  the  $50,000  class;  while 
both  video  and  PM  systems  are  estimated  to 
cost  about  $5,000  in  production.  In  terms  of 
cost,  one  would  expect  little  difference  be¬ 
tween  a  video  system  and  a  PM  system;  both 
use  a  microprocessor  computer  with  auxiliary 
source  and  detectors.  In  video,  a  light 
source  and  optical  detectors  are  used;  in  PM, 
a  sound  source  and  microphones  are  used.  As 
the  price  of  microprocessors  is  reduced,  so 
will  be  the  cost  of  PM  and  video  systems; 

$1,000  per  unit  prices  are  feasible. 

SUMMARY.  Phase  monitoring  can  be  used  for 
automated  inspection,  positioning  and  assembly 
processes.  Without  moving  parts  and  with  rel¬ 
atively  large  standoff  distances,  PM  can  de¬ 
tect  slight  differences  between  a  master  and 
another  part,  between  an  object  and  its  nominal 
position  and  between  two  objects  being  assem¬ 
bled.  The  system  is  flexible,  permitting  easy 
recalibration  for  objects  of  different  shape. 
Estimated  costs  for  the  system  are  expected 
to  be  in  the  $2,000  to  $5,000  range.  Present 
digital  equipment  can  make  several  hundred 
aeasurencnts  a  second. 

Acoustic  theory  has  been  successfully 
applied  to  the  PM  process,  resulting  in  a  high 
degree  of  understanding  of  how  geometric  changes 
are  transformed  into  phase  changes.  Accuracies 
in  detecting  changes  on  two  classes  of  surfaces 
have  been  both  analytically  and  experimentally 
investigated,  with  good  agreement  in  the  re¬ 
sults.  For  large  surfaces — areas  more  than  • 
wavelength  squared — ray-tracing  algorithms  are 
appropriate.  For  small  surfaces— areas  less 
than  a  wavelength  squared — diffraction  theory 
gives  insight  into  the  PM  process.  With  pre¬ 
sent  equipment,  changes  in  position  of  large 
surfaces  as  small  as  1  mil  (.01mm)  can  be 
detected.  The  same  equipment  can  detect  a  de¬ 
fect  in  an  object  as  small  as  an  eighth-inch 
(3.2mm)  on  a  aide. 
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NOMENCLATURE . 


a  Coordinate  direction 
y  Coordinate  direction 

•  Vector  of  phases  received  by  all  sticro- 
phones 

Phase  of  the  1th  microphone 

AS  Vector  of  phase  differences  received  by 
all  microphones 

■  *  Frequency  of  acoustic  waves 
ax  Small  movement  in  x  direction 
ay  Small  movement  in  y  direction 
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d  Deviation  or  geometric  change 

dj  Deviation  of  the  ii!2  dimension 

D  Diameter  of  cylinder 

p  Pressure 

r  Residual  error  in  measurement 

S  Sensitivity  matrix 

S'1  inverse  of  sensitivity  matrix 

ST  Transpose  of  sensitivity  matrix 

Left  minimum  inverse  of  sensitivity  matrix 
t  Time 
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